Data Management in PlinyCompute

In PlinyCompute, following data management functions are supported by PDBClient class:
(source code link: https://github.com/riceplinygroup/plinycompute/blob/master/pdb/src/mainClient/headers/PDBClient.h)

• Create Database: to create a database
• Create Set: to create a set
• Remove Database: to remove a database
• Remove Set: to remove a set
• Send Data: send a PC Object, which must be a Vector of PC Objects, to be stored in a specified set by distributing data to the cluster of nodes in a round-and-robin style
• Send Bytes: send bytes, which can be compressed, to be stored in a specified set by distributing data to the cluster of nodes in a round-and-robin style
• Partition Set: to partition a set using a PartitionComp that specifies the key extraction function
• Clear Set: to remove the content in a set without removing the set
• Flush Data: to make sure all data buffered in storage manager are written to physical storage

Storing data into an instance of PlinyCompute can be achieved by launching a client in a remote machine and connecting to a running instance of PlinyCompute either in pseudo-cluster or in a real cluster.

An example is like following:

    // creates a connection to the remote PlinyCompute instance
    PDBClient pdbClient(8108, "localhost");

    // creates a database
    pdbClient.createDatabase("example_db");

    // creates a set on the database for storing data of type StringIntPair
    pdbClient.createSet<StringIntPair>("example_db", "example_set1");
		
    makeObjectAllocatorBlock(64 * 1024 * 1024, true);
    Handle<Vector<Handle<StringIntPair>>> storeMe =
		makeObject<Vector<Handle<StringIntPair>>>();
		
    try {
       for (i = 0; true; i++) {
          std::ostringstream oss;
          oss << "My string is " << i;
          oss.str();
          Handle<StringIntPair> myData = makeObject<StringIntPair>(oss.str(), i);
          storeMe->push_back(myData);
          total++;
       }
    } catch (pdb::NotEnoughSpace& n) {
          // sends the created objects to storage
          pdbClient.sendData<StringIntPair>(
				std::pair<std::string, std::string>("example_set1", "example_db"), storeMe);
    }	

    // flush buffered objects	
    pdbClient.flushData();