1.49 00:24:05 Starting to have Borealis nodes collaborate to handle clients while reconciling their states.
2004/10/21 yna Added load_box function, to dynamically load box code.
2004/09/27 mbalazin Added a ConsistencyMngr class to the QueryProcessor. This class manages the consistency of the state of the query network in case of failures and network partitions. If you don't care about any of this, you should not see any difference. Nothing happens unless a query network uses an sunion operator.
2004/09/14 alexr Added the use of StatsMgr to the scheduler.
The statistics are being declared and used, but scheduler is not yet
collecting them at proper intervals.
Also fixed the usleep to actually utilize CPU for a box_cost
(Yanif, take note -- I will try to test this soon see if it helped)
Stats in StatisticsTools was renamed to SStats pending the namespace
reconciliation which needs to be negotiated with a few people.
2004/09/10 mbalazin When we move subscriptions automatically, we should check if destination
is self. Then we don't need to do anything.
Also modified the subscription method to return the set of successful subscriptions.
2004/09/08 dna Oops ... I only checked in one piece of my updates so some people were getting compile errors ... sorry. These updates are only to get my code to work and will be removed later. Also sorry for the rather non-descriptive function names - I wrote those really quick to get my demo to work. The next version I check in will have more useful names.
2004/08/27 mbalazin Clean-up in Admin:
- fixed a blocking call to be non-blocking in split_query
- now properly removing queries from pending once they are setup locally
(pending queries are only those in the process of being setup)
- streamlined moving load in move_load_XX. Steps still happen in series but at least now we respond to our client at the right moment.
Admin and QueryProcessor:
- added a method that changes the status of a set of queries in
on operation. That way when we move a set of queries, we delete all of
them from local query processor before checking dependencies.
2004/08/26 mbalazin Updated stream management:
- One can now subscribe to input streams. Even when no query network uses the stream.
- Moving subsets of boxes that depend on one another works (streams get forwarded correctly). Not all cominations, though.
- Updated our computations of query hunks that depend on streams. Consolidated that in query setup and deletion.
Removed stale concept of streams being "owned"... using their location instead.
Other small clean ups.
2004/08/24 mbalazin Making Anurag happy ;-) ... removed duplicate class Stream.
Most of what used to be in Borealis::Stream is now in Borealis::StreamState
The tuple buffer is now in a new class called queryProcessor/StreamBuffer/
This required updates in QueryProcessor and DataPath
Also, in DataPath, renamed all m_input_streams into m_input_paths
and m_output_streams into m_output_paths. That way, when we talk about
these streams, we will distinguish the inputs and output of query hunks
(or superoperators) and the inputs and outputs of a node. Adjusted
a few method names as well.
2004/08/24 anurag call AuroraNode's create_cpview to create cp and views
2004/08/21 yna Updated stream management to cleanly move boxes (no more intermediate streams being left everywhere).
Fixed bug in adding duplicate schemas from QueryProcessor to AuroraNode.
2004/08/19 mbalazin - Added a missing include in QOSOptimizer.h (for RPCs)
Bunch of changes mostly for connection points:
- Made CPViewDescriptions be Objects and use Name as unique id (instead of string
so that caused a few method signatures to change)
- Decoupled CPViewDescriptions from Stream
- My class Stream will go away in the meantime, I renamed the files to be MStream.* to avoid confusion in includes
- Connection points are now registered in the catalog.
2004/08/18 anurag creating cpviews being added (slowly)
2004/08/12 yna Fixed box state movement, and internal box movement.
2004/07/17 yna Added QueryProcessor::resume_queries to resume choked inputs. This is now part of the move_load process.
Added chokeSubNetwork, resumeSubNetwork, drainSubNetwork virtual functions to Scheduler. Added placeholder for drainSubNetwork prior to packing boxes.
2004/07/16 yna Admin::move_load is now a 5 (!!) stage process. Admin::move_load_2 now chokes off inputs to boxes, prior to packing them. New chokeBoxes method call defined in AuroraNode, to notify scheduler to perform cut off, and buffer.
2004/06/29 yna Added fine-grained box movement, through split_query and merge_query in Admin.
Updated movebox_test to split query into single boxes, move the boxes, and then merge boxes back into one hunk at the destination.
Added batch queue deserialization.
2004/06/25 mbalazin - When moving load, packing the state of the queries that are moved is performed in on non blocking operation
- Changed some of the RPC methods in QueryProcessor to take a copy of the query in parameter.
2004/06/24 yna Modified movebox process to remove boxes at end of transaction. Added query hunk instantiation from serialized state (queue serialization only for now).
2004/06/23 mbalazin Removing local output streams
2004/06/23 mbalazin Modified the remove_query logic to be linear.
Removing input streams from Aurora only when no more query needs them
2004/06/23 yna Fixed module build dependencies to not require QueryProcessor.
2004/06/22 yna Simple movebox implementation. Moves query hunks, but not boxes or state. Added demo, movebox_test.
2004/06/22 mbalazin Final changes on demo branch merged back into main trunk
2004/06/07 mbalazin Propagating changes from demo branch into main trunk.
2004/06/02 mbalazin branches: 1.24.2;
Modifying the we we're going to send history after checkpoints
2004/06/02 mbalazin Parameters for checkpoint messages
2004/06/02 mbalazin - Added checkpoint messages to secondary
- Bug fix in send_history -> there was an infinite loop due to brackets
- Bug fix in buffer_as_event -> was not handling the case where nothing was yet buffered
2004/06/02 mbalazin Well that's news! CVS was for once able to merge our differences without conflicts... let me test the whole thing, though
2004/06/02 jhhwang changes for passive standby
2004/05/31 mbalazin do_ha_tasks doesn't need the parameter that indicates the beginning of the interval.
2004/05/31 mbalazin Got rid of "no-ack0" and "no-trim" options.
For no-ack0, we can't don't lose tuples anyways because transmission is fast enough.
For no-trim, it was always used along with NO_HA recovery method so we can use the latter directly
2004/05/31 mbalazin When starting, the HA module tells the QueryProcessor module:
- the recovery method to use
- the primary status (true/false)... this status can change later
- the list of secondaries
2004/05/31 mbalazin Decoupling such that QueryProcessor/DataPath do not invoke methods directly on the HA module but rather send the HA module a message on its input queue.
I agree that this is extra overhead but this is the current architecture.
Given the tight coupling between DataPath, QueryProcessor, and HA we should probably put all these three classes in a single module under a single BasicComponent.
This code is not tested but it compiles... I'm going to test it now
2004/05/30 jhhwang Queue trimming works for active standby
2004/05/29 jhhwang Not a big deal.
2004/05/28 mbalazin - Replicas are now setup and created up front rather than after failure.
- Moved the RecoveryType enumeration into common/Recovery.h because both the QueryProcessor and Admin need to use that data structure.
- Changed the names of streams in the query to be shorter
2004/05/28 jhhwang in QueryProcessor,
updating stats and sending acks are decoupled.
Magda: I tested it for the ha_comparison case. Let me know if you see something wrong with your load management demo.
: ----------------------------------------------------------------------
2004/05/28 jhhwang *** empty log message ***
2004/05/28 jhhwang additions for recovery methods
2004/05/26 mbalazin Got duplicate elimination to work!
2004/05/26 mbalazin Added option to run queue trimming using only ACK1 ignoring ACK0
2004/05/25 mbalazin - bug fix: class StreamID operator != (== was working but != was not)
- added an option to disable queue trimming ("no-trim")
- a node drops input tuples if they come as a wrong version
2004/05/25 mbalazin Upstream backup simple version with ACK0 and ACK1
2004/05/25 mbalazin Playing around with simple trimming:
- Assuming a single output:
- Periodically get the last tuple in our input queue
- For each input stream,
- get the owner
- send a queue trimming message
Upstream node trims the queue by looking for a tuple
with identical content (except for the Aurora header).
Because we don't get ACKs from clients then we lose tuples if we trim at the last tuple sent downstream. So we have to acknowledge some tuple before the latest one.
Even for sigmod demo, it would be better to implement ACK0 and ACK1...
2004/05/20 mbalazin Updated query processor to support hot-swapping of input stream replicas
Updates HA module to keep track of available replicas
2004/05/18 mbalazin Bug fix!
The XML parser is not thread safe.
- Moved code that reads and parses XML config files from in thread initialization into constructors. This affected HA and NHOptimizer.
- This move required that node identifiers be set directly in the constructors rather than before starting the component's thread. I modified BasicComponent's constructor to take the node identifier as argument. I had to modify the constructor of all the Borealis components to support this additional argument.
2004/05/14 mbalazin - Removed the slow data path completely
- Merged AuroraQueryProcessor with QueryProcessor since the latter was
mainly taking care of the slow data path
- Renamed AuroraGateway to DataPath because it is more accurate
- Got rid of SimpleReader, SimpleWriter, and SimpleAdmin because
they were using the slow datapath
2004/04/14 mbalazin New dir structure
2004/06/08 mbalazin Added precaution: The periodic HA task checks that it is not getting synchronized with the stats task every time it reschedules itself.
2004/06/07 mbalazin Bug fix. Queue trimming RPC request was never returning anything.
Sorry about the frequent commits, but I need to forward my changes to the other laptop...
2004/06/07 mbalazin Ensure at most one ACK is pending.
2004/06/04 mbalazin Bug fix: when primary starts before secondary, checkpoint messages will fail. We have to refresh the endpoints to be able to connect to the secondary when it finally comes alive. We just need to be careful to reset these endpoints outside of the method that handles the response from the failed callback.
2004/06/04 mbalazin Passive standby v1.0 working!
