STAR DB API v2

STAR DB API v2

This page holds STAR database API v2 progress. Here are major milestones :

  1. Components Overview (UML)
  2. Core Components technology choice:
    1. [done] New version of XML configuration file + XML Schema for verification (.xsd available, see attachment);
    2. [done] New XML parser library :
      • [done] rapidXML (header-only, fast, non-validating parser) - will use this one as a start;
      • [not done] xerces-c (huge, validating, slow parser);
    3. [done] Encryption support for XML configuration :
      • [done] XOR-SHIFT-BASE64;
      • [done] AES128, AES256;
      • [CYPHER-TYPE]-V1 means that [CYPHER-TYPE] was used with parameter set 1 (e.g.: predefined AES key 'ABC');
    4. [done] Threads support for Load Balancer, Data preload, Cache search
      • pthreads - popular, widely used;
    5. [done] Database Abstraction Layer reshape :
      • OpenDBX (MySQL, Postresql, Sqlite3, Oracle) - API level abstraction, will require "personalized" sql queries for our system;
    6. [done] In-memory cache support [requires C++ -> DB -> C++ codec update]:
      • memcached (local/global, persistent, with expiration time setting);
      • simple hashmap (local - in memory cache, per job, non-persistent);
    7. [in progress] Persistent cache support :
      • hypercache (local, persistent, timestamped, file-based cache);
  3. Supported Features list
    • OFFLINE DB: read-heavy
    • ONLINE DB: write-heavy
  4. AOB

--------------------------------------------------------------------------------------------------------------
New Load Balancer (abstract interface + db-specific modules) :

  • dbServers.xml, v1, v2 xml config support;
  • db load : number of running threads;
  • db load : response time;
  • should account for slave lag;
  • should allow multiple instances with different tasks and databases (e.g. one for read, another for write);
  • should randomize requests if multiple "free" nodes found;

OPEN QUESTIONS

Should we support <databases></databases> tag with new configuration schema?

<StDbServer>
<server> run2003 </server>
<host> onldb.starp.bnl.gov </host>
<port> 3501 </port>
<socket> /tmp/mysql.3501.sock </socket>
<databases> RunLog, Conditions_rts, Scalers_onl, Scalers_rts </databases>
</StDbServer>