1.88 00:41:34 Making the DataPath slow down if downstream nodes cannot keep up.
2004/10/22 mbalazin Starting to have Borealis nodes collaborate to handle clients while reconciling their states.
2004/10/21 yna Added load_box function, to dynamically load box code.
2004/10/21 mbalazin Just downgraded some messages from WARN to INFO or DEBUG
2004/09/30 yna Added TableDescription object to represent tables in Borealis.
bdb tables can now be added to the queryProcessor.
2004/09/27 mbalazin Added a ConsistencyMngr class to the QueryProcessor. This class manages the consistency of the state of the query network in case of failures and network partitions. If you don't care about any of this, you should not see any difference. Nothing happens unless a query network uses an sunion operator.
2004/09/21 mbalazin - Bug fix in QueryProcessor.cc -> when creating many input streams I forgot to check if schema was already defined in Aurora.
- Added new type of union operator for my network partitions stuff... it's not done yet.
2004/09/20 mbalazin - Fixed race condition in AuroraNode.
Race was between: unsubscribing and deleting a stream (because
unsubscribe gets scheduled and executed sometime later).
- Modified "subscribe" for consistency between sub/unsub.
- Adjusted DataPath because no workaround needed anymore
2004/09/17 mbalazin Always closing an input path before sending an "unsubscribe" message to make sure we never try to enqueue anything on a stream that became an intermediate stream.
2004/09/14 alexr Added the use of StatsMgr to the scheduler.
The statistics are being declared and used, but scheduler is not yet
collecting them at proper intervals.
Also fixed the usleep to actually utilize CPU for a box_cost
(Yanif, take note -- I will try to test this soon see if it helped)
Stats in StatisticsTools was renamed to SStats pending the namespace
reconciliation which needs to be negotiated with a few people.
2004/09/10 mbalazin When we move subscriptions automatically, we should check if destination
is self. Then we don't need to do anything.
Also modified the subscription method to return the set of successful subscriptions.
2004/09/08 dna Oops ... I only checked in one piece of my updates so some people were getting compile errors ... sorry. These updates are only to get my code to work and will be removed later. Also sorry for the rather non-descriptive function names - I wrote those really quick to get my demo to work. The next version I check in will have more useful names.
2004/08/27 mbalazin Cleaned-up and simplified: local_replace_query in QueryProcessor.cc.
2004/08/27 mbalazin Clean-up in Admin:
- fixed a blocking call to be non-blocking in split_query
- now properly removing queries from pending once they are setup locally
(pending queries are only those in the process of being setup)
- streamlined moving load in move_load_XX. Steps still happen in series but at least now we respond to our client at the right moment.
Admin and QueryProcessor:
- added a method that changes the status of a set of queries in
on operation. That way when we move a set of queries, we delete all of
them from local query processor before checking dependencies.
2004/08/26 mbalazin Updated stream management:
- One can now subscribe to input streams. Even when no query network uses the stream.
- Moving subsets of boxes that depend on one another works (streams get forwarded correctly). Not all cominations, though.
- Updated our computations of query hunks that depend on streams. Consolidated that in query setup and deletion.
Removed stale concept of streams being "owned"... using their location instead.
Other small clean ups.
2004/08/24 mbalazin Making Anurag happy ;-) ... removed duplicate class Stream.
Most of what used to be in Borealis::Stream is now in Borealis::StreamState
The tuple buffer is now in a new class called queryProcessor/StreamBuffer/
This required updates in QueryProcessor and DataPath
Also, in DataPath, renamed all m_input_streams into m_input_paths
and m_output_streams into m_output_paths. That way, when we talk about
these streams, we will distinguish the inputs and output of query hunks
(or superoperators) and the inputs and outputs of a node. Adjusted
a few method names as well.
2004/08/24 anurag call AuroraNode's create_cpview to create cp and views
2004/08/21 yna Updated stream management to cleanly move boxes (no more intermediate streams being left everywhere).
Fixed bug in adding duplicate schemas from QueryProcessor to AuroraNode.
2004/08/19 mbalazin - Added a missing include in QOSOptimizer.h (for RPCs)
Bunch of changes mostly for connection points:
- Made CPViewDescriptions be Objects and use Name as unique id (instead of string
so that caused a few method signatures to change)
- Decoupled CPViewDescriptions from Stream
- My class Stream will go away in the meantime, I renamed the files to be MStream.* to avoid confusion in includes
- Connection points are now registered in the catalog.
2004/08/18 mbalazin Made calls to create_cp asynch
Some changes in control flow (all the steps remain they just happen in different order)
Changed create_cp stub in QP to pretend that creation succeeded.
2004/08/18 anurag creating cpviews being added (slowly)
2004/08/12 yna Fixed box state movement, and internal box movement.
2004/07/17 alexr
fixed some minor bugs
worked around empty box state problem
fixed up box-removing flow control
2004/07/17 yna Added QueryProcessor::resume_queries to resume choked inputs. This is now part of the move_load process.
Added chokeSubNetwork, resumeSubNetwork, drainSubNetwork virtual functions to Scheduler. Added placeholder for drainSubNetwork prior to packing boxes.
2004/07/16 yna Admin::move_load is now a 5 (!!) stage process. Admin::move_load_2 now chokes off inputs to boxes, prior to packing them. New chokeBoxes method call defined in AuroraNode, to notify scheduler to perform cut off, and buffer.
2004/06/29 yna Fixed bug in QP tracking number of dependent query hunks per stream.
2004/06/29 yna Added fine-grained box movement, through split_query and merge_query in Admin.
Updated movebox_test to split query into single boxes, move the boxes, and then merge boxes back into one hunk at the destination.
Added batch queue deserialization.
2004/06/25 mbalazin - When moving load, packing the state of the queries that are moved is performed in on non blocking operation
- Changed some of the RPC methods in QueryProcessor to take a copy of the query in parameter.
2004/06/24 yna Modified movebox process to remove boxes at end of transaction. Added query hunk instantiation from serialized state (queue serialization only for now).
2004/06/23 mbalazin Removing local output streams
2004/06/23 mbalazin Modified the remove_query logic to be linear.
Removing input streams from Aurora only when no more query needs them
2004/06/22 yna Simple movebox implementation. Moves query hunks, but not boxes or state. Added demo, movebox_test.
2004/06/22 mbalazin Final changes on demo branch merged back into main trunk
2004/06/07 mbalazin Propagating changes from demo branch into main trunk.
2004/06/03 mbalazin branches: 1.54.2;
Small things... committing to synch.
2004/06/03 mbalazin Added trimming of output queues but something strange is going on when we recovery... I need to investigate
2004/06/03 jhhwang Now the secondaries updates themselves based on checkpoint messages. Still need to work on queue trimming on backups (i.e, void update(StreamID stream_id, string last_tuple))
2004/06/03 mbalazin last_input_tuples()
and last_output_tuples()
2004/06/02 jhhwang Updates for passive standby. Needs more work yet.
2004/06/02 mbalazin Modifying the we we're going to send history after checkpoints
2004/06/02 mbalazin Parameters for checkpoint messages
2004/06/02 mbalazin - Added checkpoint messages to secondary
- Bug fix in send_history -> there was an infinite loop due to brackets
- Bug fix in buffer_as_event -> was not handling the case where nothing was yet buffered
2004/06/02 mbalazin Well that's news! CVS was for once able to merge our differences without conflicts... let me test the whole thing, though
2004/06/02 jhhwang changes for passive standby
2004/06/01 mbalazin downgraded a few messages from INFO to DEBUG
2004/05/31 mbalazin do_ha_tasks doesn't need the parameter that indicates the beginning of the interval.
2004/05/31 mbalazin ok. disabling ha completely when doing NO_HA, but NO_HA should not be the default then.
2004/05/31 mbalazin stupid extra bracket ;-)
2004/05/31 mbalazin
bug fix.
This bug comes from me getting rid of the no-trim method.
Synchronization issue. The recovery type is not set in the constructor but HA gives it to use later so we can't disable all HA activity permanently the first time that we run do_ha_tasks.
2004/05/31 mbalazin - Modified simple.cc to send the last query hunk to separate nodes as well
- One log message in query processor
2004/05/31 mbalazin Got rid of "no-ack0" and "no-trim" options.
For no-ack0, we can't don't lose tuples anyways because transmission is fast enough.
For no-trim, it was always used along with NO_HA recovery method so we can use the latter directly
2004/05/31 mbalazin When starting, the HA module tells the QueryProcessor module:
- the recovery method to use
- the primary status (true/false)... this status can change later
- the list of secondaries
2004/05/31 mbalazin Decoupling such that QueryProcessor/DataPath do not invoke methods directly on the HA module but rather send the HA module a message on its input queue.
I agree that this is extra overhead but this is the current architecture.
Given the tight coupling between DataPath, QueryProcessor, and HA we should probably put all these three classes in a single module under a single BasicComponent.
This code is not tested but it compiles... I'm going to test it now
2004/05/30 jhhwang active standby / no dups
2004/05/30 jhhwang Queue trimming works for active standby
2004/05/29 jhhwang minor fixes
plan to work on queue trimming, then duplicate elimination.
2004/05/29 jhhwang Not a big deal.
2004/05/29 mbalazin Just a precaution to avoid possible race conditions in the network partitions demo
2004/05/28 mbalazin Yet another bug fix... this time for network partitions. When we switch between replicas, we need to update all definitions first and then we should check what was the last tuple that we received. That way, if we had some tuples waiting to be enqueued either they go through and we update the definition later or we update the definition first and then we drop them.
2004/05/28 mbalazin Found the bug! It was so stupid... I had a lock and I was trying to aquire it again so I was pretty much deadlocking... and all that because of a stupid log message that I wanted to produce... it's fixed now!
2004/05/28 mbalazin Bug fix... race condition when updating catalog asynchronously so changed it to be synchronous... there's still another problem, though.
2004/05/28 mbalazin Just a tweak for debugging purposes
2004/05/28 mbalazin Just a tweak in a log message
2004/05/28 mbalazin - Replicas are now setup and created up front rather than after failure.
- Moved the RecoveryType enumeration into common/Recovery.h because both the QueryProcessor and Admin need to use that data structure.
- Changed the names of streams in the query to be shorter
2004/05/28 jhhwang in QueryProcessor,
updating stats and sending acks are decoupled.
Magda: I tested it for the ha_comparison case. Let me know if you see something wrong with your load management demo.
: ----------------------------------------------------------------------
2004/05/28 jhhwang Minor mods
2004/05/28 jhhwang additions for recovery methods
2004/05/27 mbalazin Small optimization when adding a query: adding both the new streams and the new query in one operation
2004/05/26 mbalazin Got duplicate elimination to work!
2004/05/26 mbalazin Small bug fix. The primary version was also in the list of replicas.
2004/05/26 mbalazin Added option to run queue trimming using only ACK1 ignoring ACK0
2004/05/25 mbalazin Changed log importance of queue trimming messages down to debug level
2004/05/25 mbalazin - bug fix: class StreamID operator != (== was working but != was not)
- added an option to disable queue trimming ("no-trim")
- a node drops input tuples if they come as a wrong version
2004/05/25 mbalazin bug fix. When a network partition lasts long. Unsubscribe message will timeout
2004/05/25 mbalazin - Modified subscriptions so they can specify the first history tuple that a clients desires.
- Adjusted stream buffering accordingly
2004/05/25 mbalazin Upstream backup simple version with ACK0 and ACK1
2004/05/25 mbalazin Playing around with simple trimming:
- Assuming a single output:
- Periodically get the last tuple in our input queue
- For each input stream,
- get the owner
- send a queue trimming message
Upstream node trims the queue by looking for a tuple
with identical content (except for the Aurora header).
Because we don't get ACKs from clients then we lose tuples if we trim at the last tuple sent downstream. So we have to acknowledge some tuple before the latest one.
Even for sigmod demo, it would be better to implement ACK0 and ACK1...
2004/05/24 mbalazin Avoid trying to find a replica when none exists
2004/05/24 mbalazin Updating StreamID for output streams when StreamID of input stream changes (temporary compuation for the demo not final)
2004/05/24 mbalazin Simplest version of upstream backup is working
2004/05/21 mbalazin Allowing clients to subscribe with the option to get the history replay or not
2004/05/20 mbalazin Updated query processor to support hot-swapping of input stream replicas
Updates HA module to keep track of available replicas
2004/05/20 mbalazin Added support for connection points and connection point versions.
- catalog
- subscriptions
- admin awareness
Of course, the QP doesn't know about these versions at the moment
2004/05/18 mbalazin just some logging tweaks
2004/05/18 mbalazin Bug fix!
The XML parser is not thread safe.
- Moved code that reads and parses XML config files from in thread initialization into constructors. This affected HA and NHOptimizer.
- This move required that node identifiers be set directly in the constructors rather than before starting the component's thread. I modified BasicComponent's constructor to take the node identifier as argument. I had to modify the constructor of all the Borealis components to support this additional argument.
2004/05/17 mbalazin Just tweaks of constants and messages that appear in traces
2004/05/14 mbalazin - Removed the slow data path completely
- Merged AuroraQueryProcessor with QueryProcessor since the latter was
mainly taking care of the slow data path
- Renamed AuroraGateway to DataPath because it is more accurate
- Got rid of SimpleReader, SimpleWriter, and SimpleAdmin because
they were using the slow datapath
2004/04/14 mbalazin New dir structure
2004/06/08 mbalazin Added precaution: The periodic HA task checks that it is not getting synchronized with the stats task every time it reschedules itself.
2004/06/07 mbalazin Bug fix: currently timer tasks can not be setup to fire at the exact same time. The second one will simply not fire at all... now wonder I had crazy behavior sometimes ;-)
2004/06/07 mbalazin Just a bunch of minor changes.
2004/06/07 mbalazin Bug fix. Queue trimming RPC request was never returning anything.
Sorry about the frequent commits, but I need to forward my changes to the other laptop...
2004/06/07 mbalazin Ensure at most one ACK is pending.
2004/06/07 mbalazin very minor but need to synch laptops
2004/06/07 mbalazin - Just downgrading some log messages
- Increased ha interval to 150msec
2004/06/07 mbalazin Just minor preventive fixes.
2004/06/06 mbalazin Just changing log messages
2004/06/06 mbalazin A few bug fixes:
- In Stream, it is possible that begin=end=next and buffer is empty so need to distinguish (passive standby is great at creating plenty of corner cases)
- In DataPath, when secondary takes over in passive standby, it must send history from beginning of buffer to first tuple in set of tuples currently being send (not last as I had before...)
- A few other corner small fixes in Stream
2004/06/06 mbalazin Nodes do not send anymore empty checkpoint messages or queue trimming messages.
HA demo can now run using either a stateless or a stateful query
2004/06/04 mbalazin Bug fix: when primary starts before secondary, checkpoint messages will fail. We have to refresh the endpoints to be able to connect to the secondary when it finally comes alive. We just need to be careful to reset these endpoints outside of the method that handles the response from the failed callback.
2004/06/04 mbalazin Passive standby v1.0 working!
