[aodvv2-discuss] Re: Unicast v Multicast (was System Integration)

  • From: John Dowdell <john.dowdell486@xxxxxxxxx>
  • To: aodvv2-discuss@xxxxxxxxxxxxx
  • Date: Thu, 02 Jul 2015 14:07:12 +0100

Hi all

So if can pick out some interesting points:

It's on another branch of this thread but Charlie and then Stan wrote:

<snip>


- Mark the route to OrigAddr as confirmed, but with a huge metric
and a very short timeout. This is not my preferred method, but it
would work because:
== The danger is unmeasurably small that some application
somewhere will suddenly start using the route in preference to its
previously discovered route.
==== Increasing the metric to a huge value will decrease
that small danger by yet another very large factor
== we could also prevent the huge metric route from being
supplied for any future RREQ. Doing this would reduce the danger
to be zero.


This is actually a workable, implementable notion. It's also self-contained with AODVv2, so it doesn't require additional signalling. From a WG perspective, I'd wager this stands the best chance of the 3 of surviving WG scrutiny; but may still be perceived as inferior to the MCAST approach.

Another note - as I understand it, we're talking about the time between RREQ and RREP/RREP-ACK. (To confirm the route and determine bi-directionality). That being the case, we're talking about a host route, right? I mean, we're discussing a /32 or a /128 route, back to the originator of RREQ, in order to send things *to* said originator. We're not in any way, shape or form attempting to send things *through* the RREQ originator. So John, I'm not seeing the possibility of a huge black hole - it would be vastly different if this injects a subnet route (e.g., a /8 or a /16).

</snip>

Um, so if we are simply sending the RREP to the adjacent node, indeed we can have a /32 or /128 just to that router (and in fact only addressed to that router, nowhere else), so additional traffic is unlikely. This route can remain until the end-to-end route is confirmed and possibly even until the adjacent node disappears off the interface the route points to.

However, nobody can come up with a reasonable way of making the other two options work, mostly because of the layer 3 over layer 4 problem (which IMO needs to be writ large on the slides for Prague, something along the lines of "having to use 5444 tied our hands, painted us into a corner, and left us between a rock and hard place". Maybe OLSRv2 has some secret handshake with its 5444 parser to get round this. I don't know, I'm just guessing. Might be something to quietly discuss with Henning before the meeting).

So in my view its the unicast option 3 above, or multicast.

If we're in agreement, ladies and gents, please cast your votes. I'll stand by whatever we decide.

Best regards
John

On 02/07/15 07:56, Lotte Steenbrink wrote:
Hi Charlie, hi All,

Am 01.07.2015 um 19:47 schrieb Charlie Perkins <charles.perkins@xxxxxxxxxxxxx <mailto:charles.perkins@xxxxxxxxxxxxx>>:

Hello John,

Thanks for summarizing the problem under discussion.

First, please verify my understanding:
- It is proposed to use the multicast address instead of the unicast address because multicast is in the route table and the unicast address is unconfirmed.
== Semantically, this is silly because it means you are shouting at everyone because your mechanism for whispering to your friend seems unavailable.
== The proposed solution derives from a perceived inability to have "unconfirmed" routes in the IP route table (see below).
== The danger of not solving the problem is that some application somewhere may use a route that it did not discover within a short window

I have three alternative solutions below for consideration.

First, I am pretty sure that multicast does NOT solve the problem for two reasons (which I have mentioned before):
- it is far more unreliable than unicast (10+% packet drop rates have been measured on the *wired* Internet!!), so during congested (e.g., emergency)
operations, it will fail even if unicast would almost certainly succeed
- The further enlargement by IPv6 address to limit processing by other neighbors that have been awakened will measurably increase congestion and thus
measurably further decrease reliable delivery to the intended receiver.

Here are some solutions do not add any significant overhead:
- Instead of sending RERR for undeliverable RREP, enable temporary use of the unconfirmed route (semantically, temporarily confirming it)
= after sending, mark the route as unconfirmed again until the RREP_Ack arrives

- Mark the RREQ-derived entry to OrigAddr as unconfirmed, and allow its use for RREP but DO NOT allow it for data (status quo)
== this is very sensible from a logical standpoint because we are using RREP to confirm bidirectionality
== It requires access to the IP route table to insert the unconfirmed flag
== For implementations that have such access, this is definitely a preferred solution.
==== We could make it available via a flag (e.g., IP_UNCONFIRMED_ROUTES)


That was my first choice too, but then it occurred to me that I have no idea how I would implement that (I'm not sure if it's impossible, but if it is, it would require some very dirty hacks).

When AODVv2 sends a RREP, it hands the socket API the payload and the destination address, let's call it “foo”. The network stack will then try to send this payload to the address. a process which involves asking the routing protocol (or any of the tables it supplies) “hey, do you know the next hop towards foo?”. This asking process does not involve the Network stack telling the routing protocol which payload it needs that next hop for, so the routing protocol can' tell if the Network Stack is asking about the next hop towards foo to send a RREP to foo, or if the Network stack wants to send anything else to foo (which should be forbidden). As with public tables (such as the FIB we're currently introducing), the network stack would have to have a notion of “You can use this route ONLY to send packets with a payload that looks like this”, and I don't think that's feasible either.


- Mark the route to OrigAddr as confirmed, but with a huge metric and a very short timeout. This is not my preferred method, but it would work because:
== The danger is unmeasurably small that some application somewhere will suddenly start using the route in preference to its previously discovered route.
==== Increasing the metric to a huge value will decrease that small danger by yet another very large factor
== we could also prevent the huge metric route from being supplied for any future RREQ. Doing this would reduce the danger to be zero.

Stupid question: If it's the only route towards OrigAddr, won't it still be used? It may be a crappy route, but it is a route...
(Also, there'd have to be some mechanism to buffer the “real” metric value towards OrigAddr, right? So that it can be swapped out once the route is confirmed. Which may increase the memory usage, if you have a huge/unreliable network)



Please consider these as much more reliable methods (*especially* the method that relies on kernel access, which is a very real design choice for IoT, and should be enabled).

As best I understand it, resolving this issue is NOT blocking a Last-Call-ready document.

I'd feel very bad going to last call if I knew this issue was still undiscussed. To me, it feels like breaking someone's stuff and then leaving without telling them... They'll find out eventually, and it'll be more annoying and awkward for everyone involved.

Regards,
Lotte


We will have issues arising from Last Call -- that is guaranteed. This could just as well be one of them.

Regards,
Charlie P.




On 7/1/2015 8:09 AM, John Dowdell wrote:
Hi Charlie

It's not the unicast radio performance I'm bothered about, it's the need to insert an entry in the forwarding table for all traffic in order to get the RREP sent to the next hop. The argument goes that you add the entry, send the RREP, and then wait for the Ack or the timeout. Depending on which happens first, the entry is maintained or deleted.

When the entry is made, ANY other application can choose to use the entry, believing there to be an established path to OrigAddr. This simply isn't true, since the end-to-end path is not yet fully formed. AODVv2 cannot block the other application from using the entry, because it's in the live forwarding table for all to see and use. Any traffic that does get sent this way before the route is fully formed (and actually we won't know when that is, unless we change to using end-to-end RREP_Ack) will provoke an RERR from the router where the end of the route happens to have got to, and the route will then come crashing down.

I believe this behaviour would not be exhibited if we used multicast RREPs, however ugly that might be from a wireless perspective.

The behaviour here is inside the boundary of things we as a team are charged with specifying, and thus we need to agree what is best, before the cut-off deadline.

Regards
John

On 01/07/15 06:53, Charlie Perkins wrote:
Hello John,

SMTP is an example of a protocol that does not have to be anywhere near as responsive as AODVv2, and under most circumstances not as sensitive to congestion. Our experiments with AODV (and OLSR and others) showed that the control signaling has a *crucial* effect on performance in congested networks. Congested networks are certainly within scope for a protocol developed for emergency response situations. I don't think of routing protocols as existing in isolation to the rest of the system, especially when real-time connectivity matters. If the lesson of systems integration is that the systems are all different, I don't see how it is ever possible to draw any conclusions. For ad hoc mobile wireless networks, I am confident that there is very strong lesson we must learn -- namely, to avoid congestion and unnecessary control signaling. My metric has always been performance -- if it delivers more packets, it's better. Sometimes, some packets are more important, but that's not the crucial factor at play in the circumstances under discussion. Sometimes delivering more packets has to be measured over the life of the network -- if the batteries burn out faster, the network as a whole delivers fewer packets over its lifetime. In other words, design choices can be tempered by the nature of the desired performance. I can't think of any performance factor that is improved by the suggested replacement of unicast by multicast.

I would like to hear about the dangers of unicast as you suggested in your email below. We could make a list of unicast dangers versus multicast dangers and compare.

Regards,
Charlie P.


On 6/30/2015 5:53 AM, John Dowdell wrote:
Charlie

I've been in systems integration myself for most of the last 20 years, and fully understand and appreciate where you are coming from. I do completely accept the technical validity of what you are saying, and indeed would urge these topics to be highlighted in the slides for the Prague meeting, but as per my email of Saturday, in this case we are simply not allowed to look outside the box that is called AODVv2. The piecemeal way that the manet architecture is constructed forbids it, and indeed I believe it is the case for many IETF protocols that have to exist in their own little world. Arguably the most frequently used IETF protocol, SMTP is used tens of millions of times every day to transmit email over bearers ranging from from HF radio to 10G Ethernet, but the protocol designers had to restrict themselves to their own protocol and allow the system integrators to worry about the system integration, particularly at the constrained end of bearer bandwidth.

If there is anything I've learnt about systems integration from all the systems I've helped to integrate, it is that they are all very different from each other; different use cases, different operating scenarios, different equipment fits, vastly different software loads, and in each case the communications engineering teams have had to employ different strategies to cope with presented difficulties, often acting in concert with their IT software colleagues.

It is also arguable that making AODVv2 the best and most understandable we can inside our box also allows freedom for the systems integrators, free to choose from the dazzling array of bearer optimisation technologies that are available, free to design and integrate their system to meet the needs of their customers and user communities. That means we must define the messages to say what we need them to say and no more, but at the same time they must be unambiguous and uniquely identifiable in their native form.

So with respect to your question of "Would you be happy with multicast as optional and text describing the dangers?", I believe both unicast and multicast have dangers. I believe it is our duty to point out the dangers of both methods; unicast with its declared route down which any old application can start to send, resulting in premature RERR, and multicast with its issues of radio cost (if I've understood your arguments correctly). Personally I still believe multicast is the least worst option (because of the risk of premature RERR), but we still need to achieve consensus and if I'm voted down I'll accept that.

However I have a question in return. We multicast RREQ, although out of necessity. Are there additional problems with multicasting RREP that we don't already have with RREQ?

Best regards
John

On 30/06/15 06:26, Charlie Perkins wrote:
Hello John,

Would you be happy with multicast as optional and text describing the dangers?

I consider myself to have been in systems integration for much or even most of my career, with emphasis on wireless (and parallel processing). Systems integration, to me, has never meant to ignore the realities of any particular part of the system. Far from it. Indeed, the system must be architected with full knowledge of the limitations of the parts.

Your point about the piecemeal nature of our interface to packet processing is of interest, but cannot negate the realities of what's actually happening.

Regards,
Charlie P.


On 6/28/2015 4:26 AM, John Dowdell wrote:
Charlie

With a system integration hat on, I completely agree with your point on message length and on aspects of real world operation. However I believe we must ignore that for the purposes of this draft, but raise it as an operational system consideration, so that the system integrator will consider it and take measures appropriate to their own use cases. We can do no more.

I'm happy to include text pointing out the potential system integration bear traps, but otherwise we have to comply with the piecemeal architecture that the collective drafts of manet defines.

Regards
John
------------------------------------------------------------------------
From: Charlie Perkins <mailto:charles.perkins@xxxxxxxxxxxxx>
Sent: ‎27/‎06/‎2015 16:31
To: aodvv2-discuss@xxxxxxxxxxxxx <mailto:aodvv2-discuss@xxxxxxxxxxxxx>
Subject: [aodvv2-discuss] Re: Decision time, folks

Hello John,

I did not suggest we pick any particular technology. But throughout
IETF's history there has always been consideration of reality -- and
right now most ad hoc networks that I know about run over 802.11 except
in the military which uses a lot of strange stuff.

Anyway, what do you think about my suggestion for it to be an optional
behavior with the technical considerations clearly described?

I totally disagree that message based systems are disassociated from
caring about interference. That argument leads directly to disaster and
to unusable systems -- immediately one does not care if all messages are
kilobytes in length. If RFC5444 is used to support that argument, then
we are doomed. You could say the same thing about TCP -- it's divorced
from interference. And yet no one would require all TCP messages to be
multicast with next hop specified, even though the arguments could be
made very similar to the point under discussion here.

Regards,
Charlie P.


On 6/27/2015 5:29 AM, John Dowdell wrote:
> Charlie and all
>
> I believe I have read and digested the emails on this topic and have the following to offer.
>
> 1. We haven’t specified a radio technology to use in the draft, far less considered the whys, wherefores and esoteric points of 802.11 protocols various. In the wild, I haven’t seen that many mobile ad-hoc networks built in anger, e.g. for the military or first responders, but those that I have seen have not used 802.11 technology between routers. Even some radios from a vendor with whom Stan and I are familiar didn’t use vanilla 802.11 but made a whole load of tweaks. Exactly which tweaks are a closely guarded trade secret, but the tweaks are there all the same. Acknowledging the arguments on interference, in my own home I had to move a pile of computers off 802.11 to 802.3 just to get the things to work properly, since the networks of my neighbours didn’t get on terribly well with mine and my Access Points couldn’t cope with the number of stations either. That’s before we get to the polarised viewpoints about whether 802.11 ad-hoc mode or 802.11s are actually usable or not, which I will step around for now.
>
> 2. So the reason we haven’t specified a radio technology is because we can’t. This being the IETF and not the IEEE or ETSI, we have to specify the protocol in it’s own little world. Therefore we are unable to make any assumptions about how well or badly the inter-router communication system may or may not perform. I do not disagree with the arguments presented on this list about how certain wireless protocols perform, mainly because I can’t as I do not have the depth of experience. But thanks for the information anyway and I’ll bear it in mind for other work.
>
> 3. In the brave new world of RFC5444, AODVv2 cannot send packets, nor dictate how they may or may not be formed, nor can we make any assumptions about how many packets or retransmissions may be sent. We just can’t, because we are ring-fenced into sending messages, and how the data is sent to the physical interface is no longer our concern. In fact we should be royally chastised for thinking about packets because that is the business of RFC5444 and whatever other protocols are in the path to the physical interface. If those other protocols wish to rewrite the data so it is transmitted upside down and back to front, we in AODVv2 author team cannot even so much as raise an eyebrow. Quoting from the applicability statement in RFC5444, "Packets may be unicast or multicast and may use any appropriate transport protocol or none”. I believe the whole point of RFC5444 was to provide a single path for aggregated message flows between adjacent manet routers in order to significantly reduce numbers of packets and thus collisions and retransmissions.
>
> 4. Neither can we actually take account of any discussions on how well or not a certain radio performs multicast or unicast, for the reasons explained above. I can think of many radios that will not perform multicast very well at all, but I can’t let that cloud my judgement. As an author team, we have to decide what is best for our own protocol, and the rest of the communications protocol stack both upstream and downstream does not exist, other than the edge interfaces to adjoining protocols.
>
> 5. Thus we must make the decision on whether to unicast or multicast RREP based on what is best for AODVv2 performance at the message level, and nothing else.
>
> As I understand it, if we want to send an RREP in unicast before the route is marked as confirmed, we would have to place an entry in the routing table to the neighbour just to get the RREP to send. For multicast, we can just send it. If we do multicast, I believe that means RREP-Ack must also be sent multicast. If we do send unicast, any other protocol (that we have absolutely no control over) can also use the published entry in the routing table, which could lead to all sorts of issues.
>
> I believe I have just thrown my hat in the ring for multicast.
>
> For myself, there is a point of order here which I will have to raise with the RFC5444 author team, in that how exactly are you supposed to direct a 5444 parser to send some information multicast? I’m sure that should be explained in the usability draft recently published.
>
> Regards
> John
>
>> On 26 Jun 2015, at 17:41, Charlie Perkins <charles.perkins@xxxxxxxxxxxxx> wrote:
>>
>> Hello Lotte and Stan,
>>
>> I would be willing to have this technical discussion incorporated into an optional section of the draft that explains the following:
>> - multicast RREP is optional as discussed (i.e., MAY)
>> - the problem being solved is clearly exhibited
>> - the costs of multicast (as I have listed them, and maybe more) are clearly shown
>> Then implementers can make an informed decision about whether to implement the MAY.
>>
>> I look forward to your opinion on these following points:
>>
>> The things that are most important in performance for wireless networks have to do with interference (disregarding security for a moment).
>>
>> Things like packet size do matter, but way more important is the dynamics of the control signaling. A good way to understand protocol congestion is simply to count the number of packets that have to be sent. This is typically much more important than packet size.
>>
>> So, for instance, any flooding operation has first-order effect on performance, because the total number of packets grows as n^2.
>>
>> Unreliable transmission exacerbates this problem. It is often taken into account by using a multiplicative factor of (1 + ETX), where ETX is the expected number of retransmissions before a packet is received.
>>
>> I gather that you do not dispute my points about the excess overhead and poorer reliability of multicast. These are not my opinions. These are well understood facts and have been been measured many times.
>>
>> Judging from your email, the major point of dispute lies in whether or not the problem actually occurs in practice. But this problem scenario is exactly the same for both AODVv1 and AODVv2, so AODVv1 experience is quite relevant. If you don't believe this, I would very much like to have a technical discussion about how AODVv2 is different from AODVv1 so that I can understand.
>>
>> I really feel like Stan and you are reacting to my position as simple intransigence, and I promise you that I have no such motivation.
>>
>> Regards,
>> Charlie P.
>>
>>
>>
>>
>> On 6/26/2015 8:52 AM, Lotte Steenbrink wrote:
>>> Hi,
>>>
>>> Am 26.06.2015 um 17:13 schrieb Charlie Perkins <charles.perkins@xxxxxxxxxxxxx>:
>>>
>>>> Hello Stan,
>>>>
>>>> I stand corrected. I probably shouldn't have said that -- it's way too early for me here.
>>>>
>>>> But I really can't understand why the two points I summarized for Lotte are not practically conclusive, and there hasn't been any discussion about it in the list.
>>>>
>>>> Worse performance? Why do that? Especially for a condition that almost never arises?
>>>>
>>> How do we know if it almost never arises? You cite simulation experience with AODVv1, but we have no experience with AODVv2 over real radios, in lossy networks, let alone in networks of a certain size.
>>> There is an issue, and I don't feel comfortable brushing it off with “this never happened in our simulations for AODVv1”. If we had infinite time, I'd love to investigate the entire thing more thoroughly. But we don't, so I'm inclined to go with a solution that can at least be a starting point for a discussion.
>>>
>>> To be fair, I don't have the experience I'd like to have, even though I did some implementation work. Personally, I've only come across instances of the issue when our emulation environment was acting weirdly, and it's cost me some sleepless nights. I'd love to have more reliable data and experience, but unfortunately mistakes were made, things didn't progress as planned, and I'm kicking my ass for it every day (while working to improve ;) ).
>>>
>>> Regards,
>>> Lotte
>>>
>>>> Regards,
>>>> Charlie P.
>>>>
>>>>
>>>> On 6/26/2015 8:09 AM, Ratliff, Stanley wrote:
>>>>>> -----Original Message-----
>>>>>> From: aodvv2-discuss-bounce@xxxxxxxxxxxxx [mailto:aodvv2-discuss-
>>>>>> bounce@xxxxxxxxxxxxx] On Behalf Of Charlie Perkins
>>>>>> Sent: Friday, June 26, 2015 11:07 AM
>>>>>> To: aodvv2-discuss@xxxxxxxxxxxxx
>>>>>> Subject: [aodvv2-discuss] Re: Decision time, folks
>>>>>>
>>>>>> Hello Lotte,
>>>>>>
>>>>>> Did you read my email to Justin about the difference between multicast and
>>>>>> unicast?
>>>>>>
>>>>>> To summarize:
>>>>>> - multicast is an *order of magnitude* more expensive than unicast, even
>>>>>> over a single hop with one neighbor
>>>>>> - multicast is NOT reliable over 802.11 and unicast IS reliable (so RREP is *less
>>>>>> likely* to be received).
>>>>>>
>>>>>> I don't understand how you could vote for multicast if you read that.
>>>>>> Is it just that you don't believe me?
>>>>>>
>>>>>> I doubt that Stan read it.
>>>>> Incorrect. I read it. It didn't change my position.
>>>>>
>>>>> Stan
>>>>>
>>>>>
>>>>>
>>>>>> Regards,
>>>>>> Charlie P.
>>>>>>
>>>>>>
>>>>>> On 6/26/2015 12:25 AM, Lotte Steenbrink wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>>> Am 25.06.2015 um 17:52 schrieb "Ratliff, Stanley" <sratliff@xxxxxxxxxxx>:
>>>>>>>>
>>>>>>>> John,
>>>>>>>>
>>>>>>>> My opinions on the items before us:
>>>>>>>>
>>>>>>>> 1. Unconfirmed routes: Go multicast, as was suggested by Justin Dean.
>>>>>>> +1, if that's doable considering the time we have left. Otherwise, I'd also be
>>>>>> okay with Vicky's version, but maybe we could squeeze in a sentence
>>>>>> outlining the consequences of not using rrep_ack?
>>>>>>> Concerning the Gateways, I'll trust whatever you all decide, since I don't
>>>>>> think I have the experience it takes to make a useful decision.
>>>>>>> Regards,
>>>>>>> Lotte
>>>>>>>
>>>>>>>> 2. Gateways: Leave text in about Gateway router not being default
>>>>>> router. I've not heard, therefore it's not clear, exactly where Justin & Charlie
>>>>>> are in their discussion wrt Gateways. For example, is that the only issue left?
>>>>>> Are there substantial points remaining that might adversely affect a request
>>>>>> for last call?
>>>>>>>> Regards,
>>>>>>>> Stan
>>>>>>>>
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: aodvv2-discuss-bounce@xxxxxxxxxxxxx [mailto:aodvv2-discuss-
>>>>>>>>> bounce@xxxxxxxxxxxxx] On Behalf Of John Dowdell
>>>>>>>>> Sent: Thursday, June 25, 2015 5:04 AM
>>>>>>>>> To: AODVv2 Discuss
>>>>>>>>> Subject: [aodvv2-discuss] Decision time, folks
>>>>>>>>>
>>>>>>>>> Good morning all
>>>>>>>>>
>>>>>>>>> As you all know, draft submission cut off time is 2359 UTC on Monday
>>>>>>>>> 6 July (= 0059 July 7 in UK). Vicky is on holiday next week, and
>>>>>>>>> Lotte is fast approaching (if not already in) exam time, which means
>>>>>>>>> we need to go firm on two topics currently under discussion by the
>>>>>>>>> time Vicky starts work at 0900 UK time tomorrow (so that's 0100
>>>>>>>>> Friday for Charlie,
>>>>>>>>> 0400 Stan and 1000 Lotte) in order for Vicky to make final
>>>>>>>>> substantive changes. Any remaining scraps can be tidied up on the
>>>>>>>>> Monday 6th when Vicky returns from holiday and we can submit.
>>>>>>>>>
>>>>>>>>> I believe Working Group Last Call does not mean we can't make any
>>>>>>>>> more changes. You'll have seen (and some of you have lived) the
>>>>>>>>> changes to DLEP as it went through and following the last call
>>>>>>>>> period. We still haven't had a great deal of feedback from the
>>>>>>>>> working group which means they're either busy, don't have an
>>>>>>>>> opinion, or are ok with it, but regardless we still need to plough on and
>>>>>> come to consensus for now.
>>>>>>>>> Details can still be worked on, and indeed I was just discussing
>>>>>>>>> with Vicky the prospect of an author team call while most of us are at
>>>>>> Prague.
>>>>>>>>> So the two topics are gateways and routes to neighbours.
>>>>>>>>>
>>>>>>>>> I'm not going to direct the outcome, that wouldn't be fair, but I
>>>>>>>>> need you all to come to a position of "good enough for now", to
>>>>>>>>> arrive at a document that we can all defend at Prague, and that
>>>>>>>>> position needs to be arrived at by the time mentioned above. This
>>>>>>>>> may mean each of you need to concede on certain points of detail,
>>>>>>>>> perhaps against your better judgement, but it won't be the last time we
>>>>>> can change things.
>>>>>>>>> I'm really pleased with the progress, we have come a long way. Let's
>>>>>>>>> work together to get these last points agreed (for now) and written up.
>>>>>>>>>
>>>>>>>>> Thanks and Best Regards
>>>>>>>>> John
>>>>>>>> _____________________________________________________
>>>>>>>> This electronic message and any files transmitted with it contains
>>>>>>>> information from iDirect, which may be privileged, proprietary and/or
>>>>>>>> confidential. It is intended solely for the use of the individual or
>>>>>>>> entity to whom they are addressed. If you are not the original
>>>>>>>> recipient or the person responsible for delivering the email to the
>>>>>>>> intended recipient, be advised that you have received this email in
>>>>>>>> error, and that any use, dissemination, forwarding, printing, or
>>>>>>>> copying of this email is strictly prohibited. If you received this
>>>>>>>> email in error, please delete it and immediately notify the sender.
>>>>>>>> _____________________________________________________
>>>>> _____________________________________________________
>>>>> This electronic message and any files transmitted with it contains
>>>>> information from iDirect, which may be privileged, proprietary
>>>>> and/or confidential. It is intended solely for the use of the individual
>>>>> or entity to whom they are addressed. If you are not the original
>>>>> recipient or the person responsible for delivering the email to the
>>>>> intended recipient, be advised that you have received this email
>>>>> in error, and that any use, dissemination, forwarding, printing, or
>>>>> copying of this email is strictly prohibited. If you received this email
>>>>> in error, please delete it and immediately notify the sender.
>>>>> _____________________________________________________
>>>>>
>>>>>
>>>>>
>>>
>>
>
>










Other related posts: