Wednesday, June 23, 2010

Global events considered harmful

The “global event architecture” is what could loosely be described as an architectural pattern used principally in composite applications (such as those built using the Smart Client Software Factory or, lately, Prism). It is characterized by loosely-coupled components communicating by means of broadcast events, typically marshaled through an event broker. The events are “broadcast” events in three senses: there is little or no expectation by the sender of what will be receiving the event; a given event may originate in multiple components; and the receipt of events is usually) application-wide, i.e. not confined to any particular context. Also characteristic of this pattern is that senders are connected to recipients implicitly, by the name or the type of the event.

In my article, "Global events considered harmful", I attack the global event architecture often found in composite applications. Having previously worked on three major applications that used the Smart Client Software Factory architecture or derivatives, I recently was privileged to be able to design a completely new architecture suitable for WPF applications.

By far the largest source of problems in the composite applications I had worked on is what I call the "global event architecture". The specific problems I identified were:

  • Implicit contracts are formed by the events and their arguments.
  • The contracts tend to overlap in function and meaning.
  • State is scattered throughout the application.
  • It is difficult to get a picture of the interdependencies of components, as they are implicitly coupled by the events.
  • It can be difficult to localize the events (e.g. to the scope of a single dialog or view)
  • Hanging subscriptions can cause serious application errors, and are hard to find and solve.
  • Event receipt/publication requirements in one module may force code to be written in one or more apparently unrelated modules, introducing nasty cross-cutting concerns and delocalized code.

The new architecture solved these problems, and proved to be far easier to develop on and maintain. It is based around the idea of having a "wiring builder". The wiring builder essentially writes a script to run just one application-level controller. The controller constructs containers, binds component properties (including exported commands and command dependencies), and manages workflow where necessary. From the wiring builder, we can generate dependency diagrams between components, as well as workflow diagrams indicating the transitions between application states.

The bulk of the application consists of many simple components and services, each with a single function. The components are written so that their dependencies can easily be mocked; this means that contracts between components are well-defined and testable, and also that it is easy to develop the components in isolation. This also improves reusability.

This new architecture I call the service-tree model, and it has the following advantages:

  • Having a wiring manager means that applications can quickly be reconfigured in form and function, without the necessity of a binary release.
  • The wiring manager can give you a view of dependencies and workflows that corresponds to what you would see in a functional spec.
  • It's much easier to develop the individual components, because they do not rely on external events to drive their state.
  • The code has much better locality.
  • The application configuration is centralized.
  • State contracts are explicit.

If the article seems one-sided, it's because experience with the new architecture has left me absolutely convinced that I'll never again use a non-local event. Hopefully you'll have the same experience.

Read: Global events considered harmful.

Labels: , , ,

89 Comments:

Anonymous John Gossman said...

I'm not entirely clear on your "wiring builder" (is this a code gen thing), but the "service tree" architecture is basically how Expression products (starting with Blend) are architected. The code base has a set of services and components, like a Window manager, Menu manager, Settings service etc. Each component can usually be tested by itself with mocks (okay, there are more dependencies than ideal some places). When you build a particular app, like Blend, you hardwire up an Init method that pulls in various Packages and Components and establishes explicit dependencies between them.

10:00 PM  
Blogger David Turner said...

@John Gossman: Hi John. It's extremely gratifying to know that the Expression team converged on a similar solution :-).

Our wiring builder is an interesting mix. It has a designer which resembles the old WinForms component designer. The designer generates xaml markup which actually hooks the various services up. This markup is bundled with the app settings, and at runtime we simply do XamlReader.Load to get our fully-initialized container. There are a few complications (such as passing ambient services around), but that's the essence of it.

We also found that the number of dependencies became larger in some cases than we'd want. We partially solved this by building "lambda services", which are basically just aggregators.

As I said in the post, the experience has been a breath of fresh air. This architecture is really well-suited to the WPF mindset, and it's extremely effective.

9:00 PM  
Blogger Georg said...

Hi David,

a very intresting read. But to be honest I am not sure I got your pattern. Can you supply a sample of an application using the service tree pattern? Especially of the wiring itself? Wouldn't the wiring be better handled by a dependency injection container maybe either hirachical or with local context? Maybe I simply get it wrong. But thanks for sharing it anyway.

Georg

7:47 AM  
Blogger David Turner said...

@Georg: Alas, our architecture is closed-source. However, following what John said, I reflected on the Expression codebase, and it is indeed extremely similar to what we're doing. So you could have a look there.

I do realize, based on the feedback so far, that I'll have to do a sample application. Stay tuned and I'll post it here.

7:57 AM  
Blogger John "Z-Bo" Zabroski said...

Passing around ambient services sounds a bit gross. I prefer an object capability model, since it allows for fine-grained composition of services and reifies the object graph as a security graph, thus forcing you to consider partial failure. I don't understand why OO programmers fight this model.

I had a much more detailed reply concerning your comments about "events with arguments" and "hanging subscriptions", but Blogger 404'd me.

I'll try to summarize, since I can't bear to spend another 30 minutes typing up an elaborate position.

In short, I think you are overgeneralizing pubsub systems and not really discussing model of computation here, and let me be clear, model of computation is what component communication is all about! Beating Prism's "global event architecture" is not hard, since I am not aware of it being based on any theoretical foundations! For example, CAR Hoare argues in his book Communicating Sequential Processes that he prefers a synchronous model of communication with unbuffered connections, because fixed length buffers have some thorny deadlock problems. How is THAT for hard reasoning about "hanging subscriptions"?

10:45 PM  
Blogger David Turner said...

John Zabroski, it sounds like you have a good background on this. We may have several points of agreement here, and I'd like to discuss it further. It's was a bit cheeky of me to use the title I did on what is not a very academically-minded article. But I do take your points and agree.

2:12 PM  
Blogger John "Z-Bo" Zabroski said...

Actually, "Considered Harmful" papers are "Considered Harmful" in academia. Even the first one was not originally titled that ("Goto Statement Considered Harmful"). Gof author Ralph Johnson has a Ph.D. student Jeff Overbey who had a funny take on this. Google for "considered harmful considered harmful".

Overall, I liked how you attempted to break down your intuition into commonly accepted OO design wisdom (esp. contracts, orthogonality, atomicity). However, I think you fall short in your solution. From what I can tell about your wiring builder, it does not ensure that events are routed through the correct layer of the domain. Instead, you appear to be hijacking dependency properties to build a data dependency graph. In general, I don't like this because I personally find it hard to debug, because inserting new components equates to inserting new sequents on a lattice, which can change the entailment relation of that lattice.

I also think you're conflating component definition with component coordination. WPF has this same foul smell. To me, components should communicate via a special coordination language. Best argument for this I've ever read is Edward A. Lee's paper "The Problem With Threads". A formal coordination system, like the .NET 2.0 Concurrency & Coordination Runtime for robotics studio, or even F# agents, is a much better way to coordinate communication than to use WPF's awful component object model.

Cheers,
Z-Bo

11:41 PM  
Blogger John "Z-Bo" Zabroski said...

Ergh, my lattice explanation was too nerdy. Let me change it: This is commonly known in OO as the fragile base class problem. Some people even call it the Fragile MIDDLE class problem, because research shows that when people maintain inheritance hierarchies, they most often check the base class and the deriving class, but not any inbetween classes. A Fragile MIDDLE class therefore can be thought of as a sequent on a lattice. By changing an inheritance hierarchy, all client classes that use the middle class will break. This has a whack-a-mole bug feeling.

11:54 PM  
Blogger David Turner said...

Re considered harmful considered harmful - that was sort-of the point :-).

Yes, I'm doing the hijacking you describe, and yes, it sometimes results in issues characteristic of non-orthogonality as data and events mix. However bear in mind that, in the WPF model, commands are [can be] properties. The command-routing is actually handled slightly differently in our for-real wiring builder. I did allude to generating workflow documents in my OP. Not quite what you're talking about, but very practical from the point of view of standard bespoke development processes.

I suspect that a "special coordination language" might be a step too far for the moment. I do see harbingers of these ideas reaching the mass market, but for now UIs live in a fundamentally one- or two-threaded world. Thanks for the link to Lee's paper; I hadn't read it and it is indeed excellent.

Now we come to fragile base classes. For a long time, I've considered classical inheritance hierarchies to be upside-down. I started writing a paper on the subject, but had to abandon it due to time constraints. Essentially, I view objects as a coalition of aspects. For example: Fido is a dog. Dogs characteristically have a bark; transitively, they are vertebrates and therefore have spines. However it is only accidentally the case that Fido is called Fido; he has a name in respect of being a pet, which is nothing to do with doggishness.

Consequent to the above, I rarely use contractual base classes, and when I do it's usually for convenience.

My perspective on the middle-class problem as it applies to wiring diagrams is this: if it's a problem, you haven't respected atomicity and/or orthogonality. Does that address your point?

9:24 AM  
Blogger John "Z-Bo" Zabroski said...

What you typically find is that software engineers push out bigger and bigger systems. In the 1990s, it was COM, and it failed. It was replaced with .NET.

.NET 3.0 is similar to that COM and DCOM experiment we saw in the 1990s. What ultimately happened with .NET 3?

- MSFT customers told WPF architects that WPF has some really complicated features. "Designers" especially had a hard time understanding WPF's notion of time

- No MSFT customers really used 90% of WCF, because it required a Ph.D. to use successfully. The Mono project initially focused its efforts on the REST stuff, because that is what people could understand.

- Windows Workflow Foundation (WF) was burned to the ground from practically even before .NET 3 was released. The new WF in .NET 4 bears very little resemblance to the one in .NET 3.

I'll focus on explaining how .NET 3 is pushing for more formal methods on the .NET platform... specifically WPF.

In WPF, it has a resource location subsystem, a synchronous reactive ("data binding") subsystem, a system of time-continuous and time-discrete with no good story for how piece-wise partitioning of time-varying functions work (Silverlight has improved this by de-emphasizing the role of triggers), and separation of control from data. In WPF, these subsystems really aren't orthogonal and they are intertwined and inter-dependent on one another, creating what scientists call "cyclomatic complexity". In short, it was probably ridiculously hard for the WPF team to properly unit test their code.

As for this comment:

@My perspective on the middle-class problem as it applies to wiring diagrams is this: if it's a problem, you haven't respected atomicity and/or orthogonality. Does that address your point?

Well, sort of. But when the CEO of a Fortune 500 company yells at you and your best explanation is you forgot to respect atomicity and/or orthogonality, you learn that you want more and more confidence that you did respect those properties. This is the danger in a wiring system that depends on the semantics of things "by its name" (reflecting on its nominal type) rather than by its (algebraic) structure. For example, how do you debug a WPF Data Binding if the WPF reactive subsystem is using a different DataContext=? and Binding Name=? than you expected? In my experience, the best way to solve these problems is to avoid them.

What I think you see is that simple wins. Silverlight is way simpler than WPF in the aspects that I hate about WPF. A lot of this can be contributed, IMHO, to Conway's Law - "...organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." For example, even in .NET 3, John Gossman's Blend team basically overrided everything the WPF controls team was doing, and replaced each control definition with 1100 lines of Blend-default XAML. When Silverlight came around, John was picked to define what parts of WPF made it into the 4 MB CLR core. The fact that his Blend team no longer had to fight against the grain - and that Microsoft's organizational structure better fit the needs of its customers (integrated Silverlight/Blend story) has produced a MUCH better product.

But Conway's Law can go the other way - in .NET 3, the notion of a DependencyObject and DependencyProperty was propagated across FX subsystems, and these types did not directly match. The primary reason for this was that the reactive subsystem for data binding was used as a catch-all for all kinds of framework-specific hacks (such as WPF using it for quenching and coalescing events). My best guess is that the development structure for .NET 3 and the FX libraries didn't really make it convenient to properly design these subsystems. It's not like MSFT hires dumb people - they're all smart. So the problems usually come down to structure, in my experience.

5:12 PM  
Blogger David Turner said...

I do know what cyclomatic complexity is :-). Yes, it's bad in WPF, although possibly not as bad as in various other parts of the .NET framework assemblies (I'm thinking of System.Xml and the WinForms binding implementation in particular).

And simple absolutely wins almost absolutely.

You said: "But when the CEO of a Fortune 500 company yells at you and your best explanation is you forgot to respect atomicity and/or orthogonality, you learn that you want more and more confidence that you did respect those properties."

I'm not arguing against that in the slightest. My work is aimed ultimately at formalizing the representation of these properties of software systems. The example I provide expects explicit wiring to be specified; I had in mind a circuit-diagram sort of notation for this. So it very much depends on how you generate the configuration, in my view.

Something that's stuck with me for a long time is a response Alex Stepanov gave in an interview with STLport: "I find OOP technically unsound. It attempts to decompose the world in terms of interfaces that vary on a single type. To deal with the real problems you need multisorted algebras - families of interfaces that span multiple types.".

5:49 PM  
Blogger John "Z-Bo" Zabroski said...

@And simple absolutely wins almost absolutely.

Pretty much. One thing you'll notice is that soon after .NET 3, the Microsoft PDC presentations changed their vocabulary tremendously. I don't know if anyone did a study on this, but I would bet in 2007 the word "simple" was never said, and in 2008 it was said like a billion times. The pitfall with labeling things as "simple" though is that sometimes you end up building the next TCL: a subsystem that is really just a glorified scripting language.

I am not really that familiar with System.Xml or WinForms. I grew up writing a lot of Java code in college, and when time approached to get a job in the real world, I was extremely horrified by stupid and ugly books like "J2EE Design Patterns", which would make any mathematician want to vomit - in those fake design patterns, object responsibilities are scattered throughout the system, and the entire system is flattened. J2EE practices always struck me as a bad idea, and so I resisted taking any job in enterprise Java development - finally I found the place where I work now. They mainly use me here for my jack of all trades - master of none skillset. So I've dabbled in just about everything.

7:24 PM  
Blogger John "Z-Bo" Zabroski said...

I just looked up System.Xml, and saw XPathNavigator mentioned on the web as a class with high cyclomatic complexity [1].

I compared .NET 4's implementation to Silverlight's, and noticed that Silverlight has stripped out the Cloneable logic.

In general, when I refer to cycloamtic complexity, I am really using it as a measure of faithfulness to run-to-completion (RTC) semantics. One of my big principles is that an object should not go out and get its dependencies, but WPF does that a lot - stuff like DataTemplate in WPF delegates to the WPF resource subsystem for things like the DataType property (if not null). Here, it is not merely the +1 for the check to see if DataType property is null or not. We also have to take into account that the reason this +1 is occurring is because the component (DataTemplate) doesn't know how to fulfill its own responsibilities! Silverlight, by the way, got rid of this god awful DataType property [2]. Just one of many examples of the great job the Silverlight team did cleaning up the WPF mess. RTC Semantics are not preserved by DataTemplate, so it is best in my books not to use it at all. Best to avoid code where the authors couldn't be bothered to write it well enough to specify what happens when its component parts fail.

[1] http://devreflection.wordpress.com/2007/06/11/cyclomatic-complexity-and-refactoring/
[2] I've seen articles on MVVM that advocate using DataType property in WPF. I don't understand this, since it is really just fancy reflective binding sugar but the cost of this sugar causes cancer of the semicolon (you now are dependent on the WPF resource system correctly resolving your syntactic name references - how do you unit test this and what do you do if WPF has a bug or works a way you don't like, you're screwed).

12:06 AM  
Blogger David Turner said...

I had the unfortunate experience of reimplementing XPathNavigator (along with the rest of the Xml.* namespaces).

I have used DataType-d data templates quite successfully in heterogeneous lists and where I need to vary the view based on the data (for example, line chart vs column chart). This is really shorthand for implementing a template selector, which would otherwise be tedious. Since the data templates are invariably defined in-place and the view-models backing them are generally internal classes closely tied to the primary view-model, I don't consider this to be bad design. It's certainly a time-saver.

Where things start to get really hairy is with resource lookups ten steps distant, and yes, bindings that span elements and data contexts.

2:36 PM  
Blogger John "Z-Bo" Zabroski said...

@This is really shorthand for implementing a template selector, which would otherwise be tedious.

EXCEPT that with a template selector, you can base your selection criteria on a "well-founded set" as your data source. You could just as easily write a DataTemplateSelector that implements DataType-like logic, except: (a) it wouldn't need to waste clock cycles doing a recursive ascent search through the visual/logical tree and WPF resource subsystem [Why do you think ugly things like AncestorBinding exist, as well as other special Binding forms?] (b) if you want to model the types in your system as "open" and allow people to add more types, you can, modularly; as your requirements change, the places in which you must make changes remains related to the structural changes needed in your gross application structure, reflected by real-world requirements. (c) this way is much easier to unit test, since you are not dependent on WPF

7:50 PM  
Blogger David Turner said...

Yes-no. Not everything needs to be extensible; and it hardly seems worth testing the DataTemplateSelector you mentioned, when you have a known-good one already. Given that the templates are defined locally, the up-tree search hits immediately, so the overhead is (except in some extreme cases) insignificant.

Which leaves us with the case of an open type system, in which case I agree with you; the loss of locality between type and template is not justified by the convenience (if there is any) of using the built-in resource mechanism.

8:06 PM  
Blogger John "Z-Bo" Zabroski said...

It has nothing to do with where you define a DataTemplate. It has everything to do with how it resolves DataContext and a Typed resource in a WPF Resource Dictionary.

I think the disagreement here is that you think WPF DataTemplate is good. I am saying it is Mickey Mouse programming, like Routed Commands. I can go into more detail on this, discussing parametric polymorphism and problem domain abstraction and why the goal of OO should be to externalize unknowns to produce deterministic inner modules, and why this makes for easier testing and higher quality assurance, (a) if you'd like (b) I find the time/energy for a coherent presentation.

8:31 PM  
Blogger John "Z-Bo" Zabroski said...

Let me try this another way.

In WPF, if you use DataTemplate DataType, then you are specifying a wide contract - the values returned by this interface are NON-DETERMINISTIC because the input/output relation of the interface is dependent on the current state of a myriad of factors, all related to WPF resource handling. You can not know at compile time what to expect, including partial failure. And because this mapping is implicit (using reflection), you cannot guarantee that somebody else won't write code that injects some matching CLR object inbetween, or what to do if the only CLR object matching it is removed from the resource system. DataTemplate doesn't even let you specify what to do in such failure scenarios. It is stupid and ugly. There is even a Triggers property, but the Triggers subsystem in WPF is a bit whacked too, such that you can't use Triggers to detect intermittent resource failure.

Let me emphasize you really, really want determinism when doing programming. That is one reason why so-called object-functional languages like F# and Scala are becoming more popular. Determinism is probably the second most important property a system can have, after faithfulness of problem domain abstraction. Correct abstraction, then determinism.

8:44 PM  
Blogger David Turner said...

I'm not saying it's good, I'm saying it's efficient - from the perspective of development costs. 90% of the time, the terrors you refer to don't apply. So not the right mechanism to develop shuttle guidance systems; but definitely right for your average CRUD/BI app.

Also, the matching that occurs most certainly is deterministic. We haven't got quantum computing just yet. What you mean to say is, it gets complicated. True enough, you can concoct wildly complicated inferential systems with it. The point is, you don't.

9:50 PM  
Blogger John "Z-Bo" Zabroski said...

Well, I am the designer of a BI app - actually, it is working towards a full-fledged VM, since I believe the notion of BI apps should be completely virtualized. Partial failure and an object capability model has proven really handy for building BI apps, in my experience.

I'm not sure why you think DataTemplate DataType=? is simpler than writing a similar data template selector. Most .NET IoC containers these days provide a pretty robust resource querying language for determining what resource to load - although they could be much better designed if programming language researchers tackled the problem. Most IoC containers support features like "Contextual Binding", "Profiles", and "Autowiring". The difference is my change makes your UI subsystem more deterministic, since it is no longer looking for dependencies in the WPF resource subsystem (which is tightly coupled to the UI subsystem). Instead, it gets its dependencies from external observers. This means you can convert the UI description itself into a dataflow graph, and any in situ edits to the state of the dataflow graph correspond to editing nodes in the dataflow graph. That's a way simpler mathematical model.

@So not the right mechanism to develop shuttle guidance systems; but definitely right for your average CRUD/BI app.

I am not building command-and-control military systems, although I know a lot about that subject area and how they could be used e.g. mostly from watching too many war movies and knowing the technology exists today to make most of that sci-fi stuff a reality. :)

10:28 PM  
Blogger David Turner said...

My point is that writing a data template selector that basically does "if (type == x) return FindResource("xTemplate")" is a waste of time. Just another class that replicates built-in behavior and needs to be tested. If the set of types is predetermined and fixed, what does it buy you?

11:33 AM  
Blogger John "Z-Bo" Zabroski said...

It buys you a lot.

1) Easier portability to Silverlight (I am disapointeded that Microsoft's Composite Application Guidance doesn't mention this as a best practice)

2) If you write it in F#, it will be more terse, because you can take advantage of partial application and active pattern matching to deconstruct/destroy an aggregate object into its component parts. F# can also provide you with static guarantees about your pattern matching, such as completeness.

3) You don't have to use FindResource, because you don't have to store the resource in a WPF resource dictionary.

4) Determinism+Partial Failure. Certaintiy is really _the abstraction of uncertanity_. If we cannot provide a DataTemplate, then we can specify what to do if we can't. In other words, we can handle partial failure by deterministically specifying what to do when what we expected to happen, didn't happen. In my applications, this could include 'fallback' DataTemplates when more advanced visualizations cannot be found - sometimes we fetch visualizations from a visualization repository, so that users can mash-up visualizations and share them with other users.

5) We can now use a formal technique for specifying interface template construction, using abstract data types (ADTs), called Formlets.

6) ... I am sure there are more reasons, but I feel like an accountant with "accountant bulleting disease" making this list.

4:33 PM  
Blogger John "Z-Bo" Zabroski said...

By the way, David, as I don't have your email, I want to thank you here for being so willing to openly debate with me. Your insistence on asking for really good explanations is appreciated. Most people either (a) give up and think it is too hard to understand (b) don't even bother

You forced me to put into words some things that I haven't communicated to others before. I think I'll write a blog post write-up about this. I've written a bit about DataTemplates before, but I think I'll write another about why FindResource is pure evil as well, and then connect the two ideas.

5:39 PM  
Blogger David Turner said...

Dee kay turner at gmail. Happy to discuss in more detail there.

7:08 PM  
Anonymous Anonymous said...

buy tramadol online do people buy tramadol - buy tramadol hydrochloride

8:49 AM  
Anonymous Anonymous said...

xanax online generic equivalent of xanax - xanax high blood pressure medication

6:56 AM  
Anonymous Anonymous said...

xanax online xanax xr vs generic - 2mg xanax not working

6:50 AM  
Anonymous Anonymous said...

cheap tramadol online buy tramadol online pharmacy - order tramadol online safely

11:22 AM  
Anonymous Anonymous said...

buy tramadol online tramadol dosage compared to vicodin - buy tramadol with paypal

1:58 PM  
Anonymous Anonymous said...

buy carisoprodol carisoprodol soma scheduled drug - soma carisoprodol 250 mg tablets

6:45 PM  
Anonymous Anonymous said...

buy tramadol online tramadol 750 - tramadol dose get high

2:21 AM  
Anonymous Anonymous said...

generic xanax xanax drug side effects - what are xanax pills prescribed for

3:39 AM  
Anonymous Anonymous said...

buy tramadol online tramadol for dogs reviews - tramadol hcl good

5:56 AM  
Anonymous Anonymous said...

carisoprodol 350 mg carisoprodol show up drug test - carisoprodol gluten free

10:01 AM  
Anonymous Anonymous said...

buy tramadol online tramadol hcl is it addictive - tramadol hcl withdrawal

3:07 PM  
Anonymous Anonymous said...

buy tramadol online ultram side effects weight loss - tramadol withdrawal runny nose

4:27 PM  
Anonymous Anonymous said...

carisoprodol 350 mg carisoprodol dosage information - carisoprodol orange juice

5:04 PM  
Anonymous Anonymous said...

xanax online ways get high xanax - buy fake xanax bars

6:58 PM  
Anonymous Anonymous said...

can i buy xanax online xanax 1 mg color - 10mg xanax generic

2:33 PM  
Anonymous Anonymous said...

buy tramadol online tramadol with high blood pressure - can i buy tramadol over the counter in the usa

6:52 AM  
Anonymous Anonymous said...

buy carisoprodol carisoprodol 350 mg tablet side effects - carisoprodol 350mg tablets price

6:49 PM  
Anonymous Anonymous said...

xanax online 1.5mg xanax and alcohol - cost of generic xanax at walmart

1:29 PM  
Anonymous Anonymous said...

buy carisoprodol carisoprodol 350 mg buy online - carisoprodol blood pressure

11:19 PM  
Anonymous Anonymous said...

buy tramadol online tramadol 0 1g - tramadol 50 mg whyte pill

12:02 AM  
Anonymous Anonymous said...

carisoprodol 350 mg buy soma carisoprodol online - carisoprodol 2410

2:08 PM  
Anonymous Anonymous said...

buy cialis online cialis online usa no prescription - buy cialis online american express

4:57 PM  
Anonymous Anonymous said...

cialis online cialis online discount - cialis что это

12:06 AM  
Anonymous Anonymous said...

xanax online xanax side effects weight - xanax withdrawal symptoms timeline

6:15 AM  
Anonymous Anonymous said...

buy cialis online cheap cialis ireland - cialis super active

9:41 AM  
Anonymous Anonymous said...

buy cialis no prescription buy cheap cialis today - cialis prices usa

2:51 PM  
Anonymous Anonymous said...

buy cialis online cheap cialis without rx - buy cialis levitra

2:50 AM  
Anonymous Anonymous said...

cialis online pharmacy order cialis daily use - cialis reviews webmd

12:50 AM  
Anonymous Anonymous said...

cialis online buy viagra cialis online usa - cialis online in u.k

8:47 PM  
Anonymous Anonymous said...

xanax online does xanax show up swab drug test - xanax 2mg yellow bar

1:10 AM  
Anonymous Anonymous said...

buy tramadol tramadol online without prescriptions - buy tramadol online cheap

3:54 PM  
Anonymous Anonymous said...

learn how to buy tramdadol tramadol online 99 - tramadol 50 mg get you high

1:03 AM  
Anonymous Anonymous said...

http://landvoicelearning.com/#51438 tramadol kick in time - tramadol 50 mg for anxiety

7:41 AM  
Anonymous Anonymous said...

learn how to buy tramdadol tramadol withdrawal leg pain - tramadol 800mg

3:07 PM  
Anonymous Anonymous said...

tramadol 100mg tramadol hcl usp monograph - but tramadol online usa

1:05 PM  
Anonymous Anonymous said...

buy tramadol tramadol online without prescriptions - buy tramadol online cheap

12:04 PM  
Anonymous Anonymous said...

buy tramadol online tramadol 50 mg for humans - tramadol er 100mg tablets

9:50 PM  
Anonymous Anonymous said...

where to buy tramadol online tramadol online safe - tramadol iv

2:43 AM  
Anonymous Anonymous said...

buy tramadol online cheap tramadol vs norco - buy tramadol 150 mg

4:27 AM  
Anonymous Anonymous said...

buy tramadol tramadol for dogs price - tramadol 50 mg and xanax

6:27 PM  
Anonymous Anonymous said...

http://landvoicelearning.com/#51438 tramadol 50 mg uses side effects - buy tramadol in egypt

12:38 AM  
Anonymous Anonymous said...

buy tramadol side effects of ultram tramadol - tramadol in dogs

5:02 PM  
Anonymous Anonymous said...

http://buytramadolonlinecool.com/#30807 illegal get tramadol online - buy tramadol online visa

12:47 AM  
Anonymous Anonymous said...

buy tramadol is tramadol online real - buy cheap tramadol overnight

8:22 PM  
Anonymous Anonymous said...

http://blog.dawn.com/dblog/buy/#34852 tramadol drug buyers - tramadol 50 mg online pharmacy

3:17 AM  
Anonymous Anonymous said...

http://reidmoody.com/#54126 ativan how does it work - can you buy ativan mexico

6:05 PM  
Anonymous Anonymous said...

buy ativan online ativan 6mg - ativan 0.5mg addiction

10:53 PM  
Anonymous Anonymous said...

http://reidmoody.com/#54126 ativan withdrawal vertigo - ativan vs xanax stronger

10:11 PM  
Anonymous Anonymous said...

http://reidmoody.com/#36921 ativan get you high - ativan online no prescription canada

2:57 PM  
Anonymous Anonymous said...

buy xanax online pills that look like xanax bars - xanax and alcohol teenagers

7:13 PM  
Anonymous Anonymous said...

buy tramadol online buy tramadol online no prescription usa - tramadol withdrawal oxycontin

8:55 AM  
Anonymous Anonymous said...

buy tramadol online tramadol 50 mg for migraine - help with tramadol addiction

3:21 PM  
Anonymous Anonymous said...

buy tramadol india tramadol online visa - how tramadol addiction is treated

1:28 PM  
Anonymous Anonymous said...

buy xanax online xanax withdrawal diarrhea - xanax effects dopamine

3:11 PM  
Anonymous Anonymous said...

http://staam.org/#90723 cheap 100mg tramadol - tramadol overdose reversal

6:41 AM  
Anonymous Anonymous said...

http://ranchodelastortugas.com/#71453 buy xanax online cod - took 2mg of xanax

2:09 AM  
Anonymous Anonymous said...

buy xanax online xanax for anxiety symptoms - buy xanax bars online

11:13 PM  
Anonymous Anonymous said...

buy xanax online pass hair drug test xanax - percocet xanax drug interactions

3:16 PM  
Anonymous Anonymous said...

buy xanax online xanax generic names - xanax bars for sale online

10:24 AM  
Anonymous Anonymous said...

buy xanax online order xanax online legally - xanax side effects for pregnancy

12:08 AM  
Anonymous Anonymous said...

xanax no rx order xanax india - fake xanax 1mg

7:20 PM  
Anonymous Anonymous said...

buy xanax online xanax and alcohol webmd - xanax overdose long term effects

9:17 AM  
Anonymous Anonymous said...

Huvhjg [url=http://hermeskelly.finniwolf.com][b]hermes sale[/b][/url] hermes olde english bulldogge http://hermeskelly.finniwolf.com Odbxhe Dziojo hermes handbag

6:30 AM  
Anonymous Anonymous said...

Your currеnt article offеrs eѕtabliѕhed useful to me.
It’s very educatіonal anԁ you really arе natuгally νerу experiencеd in this field.
You gеt poρped my ρersonal eyеs to be ablе to numerοus οріnion of this
topic with іntriquing, notablе and sоlid content.


Take a look at my web ѕite ... Buy Valium

12:49 PM  
Anonymous Anonymous said...

order tramadol overnight buy tramadol medication - order tramadol online mastercard

2:46 PM  

Post a Comment

<< Home