Subject: Re: [secdir] Secdir review of draft-ietf-sip-fork-loop-fix-07.txt

From: Cullen Jennings <[email-masked]>
Date: Fri, 17 Oct 2008 13:25:57 -0600

Hi All,

I have put this draft on the agenda for next weeks IESG call but I  
have not seen much discussion of what if any changes should be made  
given the review below. The unfortunate problem is we don't have any  
particularly good solutions for the loop issue and mostly need to  
decide what we can get published and implemented that is significantly  
better than 3261.

Cullen

On Sep 29, 2008, at 12:11 PM, Charlie Kaufman wrote:

> I am reviewing this document as part of the security directorate's  
> ongoing effort to review all IETF documents being processed by the  
> IESG.  These comments were written primarily for the benefit of the  
> security area directors.  Document editors and WG chairs should  
> treat these comments just like any other last call comments. Feel  
> free to forward to any appropriate forum.
>
> Since my last review of this document where I complained of a  
> dangerous exposure to looping attacks caused by malicious - or even  
> accidental - misconfiguration, the document has been revised so as  
> to make such loops somewhat less likely to occur by accident and  
> somewhat less disastrous for network loading when they do occur. The  
> attacks and their consequences are also much better explained in the  
> document. Unfortunately, they don't solve the fundamental problem of  
> having a single request generating an essentially infinite load. It  
> still seems unacceptably risky to deploy.
>
> The problem can be compared to mailing list loops. If two Internet  
> mailing lists each contain the other as a member, any message sent  
> to either list ends up cycling between the two list processors  
> sending continuous streams of messages to other members of both  
> lists. In the Internet's younger days, that happened routinely. If  
> three Internet mailing lists each contain the other two lists as  
> members, and if the expanders are smart enough to send out copies of  
> the message in parallel to the multiple recipients, then not only do  
> other members get a continuous stream of messages but the list  
> expanders will be overwhelmed by all of the parallel requests. I  
> don't know exactly what solutions we have found to this problem, but  
> it's possible it could be retrofit onto SIP.
>
> SIP is like the mailing list expander except that doesn't deposit  
> messages in anyone's mailbox. Instead, it is searching for  
> 'presence' of someone or something so as to be able to direct an  
> instant message, phone call, or some such. Each place where it looks  
> for someone may (like a mailing list) forward the query onto a  
> number of other SIP servers. SIP does detect loops, but in remains  
> the case that if there are N places to look for someone, and each of  
> those N places points to the other N-1 places (a not unlikely naïve  
> configuration), there will be N! queries processed before the search  
> terminates. For N of even moderate size, that can effectively be an  
> infinite loop.
>
> The main improvement in loop detection from the previous version of  
> the spec is that it limits the parallelism of the processing of  
> these N! queries so that a single bad query cannot deny service to  
> other users of the SIP server. With the sample "Max-Breadth" of 60,  
> it probably does not take a very large number of these infinite  
> loops to overwhelm the system.
>
> The proposal I made in my last review which limited the total number  
> of queries in the processing of a single request would break too  
> many real-world scenarios. Other possibilities were discussed on the  
> SIP mailing list and this was apparently the consensus approach.
>
> I don't believe there is any complete solution to this problem that  
> doesn't break some known desirable scenarios. I can think of some  
> partial solutions that improve the situation - perhaps be enough,  
> but I can't be sure. These include:
>
> When checking for loops, test not only against earlier points in the  
> history of this query thread, but also against any parallel queries  
> that are currently waiting for responses from other SIP servers.  
> This would more quickly catch the overload problem because in those  
> circumstances the tables of delegated queries would quickly explode.
>
> Any time a loop is detected, the forwarding table entry that caused  
> the loop should be disabled and flagged for an administrator to  
> examine. This could be done by suspending the identity being  
> searched for, or if enough information is present only suspending  
> the forwarding address that generated the loop. The upside of this  
> scheme is that loops will be broken quickly. The downside is that it  
> creates a new security exposure where someone intentionally 'fakes'  
> a loop in order to make some entity unfindable.
>
> There should be a real time bound on any forwarded SIP request,  
> after which the request is deleted and its state removed. I don't  
> know the appropriate real time bound, but I can't imagine any reason  
> to let a request bounce around in the network for more than a week.  
> One example query in the document (section 3) was estimated to take  
> 3 trillion years to complete.
>
> Smaller issues:
>
> There should be some discussion in the document about what servers  
> should do in the case of overload. If particular, crashing is  
> probably not a good idea; overload should be expected. If the table  
> of outstanding forwarded requests overflows (as it will in error and  
> some non-error cases), there is a trade-off between discarding the  
> oldest request, the newest request, the most forwarded request, the  
> least forwarded request, etc. When the capacity for incoming  
> connections is exceeded, it may be better to close some existing  
> ones to allow new ones to arrive.
>
> Section 4.2.4: The document suggests a way to make loop detection  
> cheaper by using a hash function over the repeating data and  
> comparing the hashes. The original document suggested MD5. Even  
> though for the purpose of this document MD5 would be perfectly  
> reasonable, MD5 has become politically incorrect so the document now  
> recommends CRC-32c as an alternative. There are two problems with  
> that. One is that 32 bits are arguably not enough. Failing 1 in 4  
> billion requests is not very many, but it will happen. As the  
> forwarding lists get longer (and particularly if a check is made  
> against outstanding requests) the failure rate will get much larger.  
> More subtly, CRC-32c is 'linear', so if there is a collision causing  
> a failure, it may be a solid failure where retries also fail. A  
> better solution would be to propose a politically correct hash (like  
> 64 bits of a SHA-256) and put a hint/nudge that MD5 (or even MD4)  
> would be just fine and explain why. This is only an implementation  
> hint - there is no requirement that different implementations do  
> this the same way. The best solution here is to compute a relatively  
> small hash as a performance hack, but when there is a match check  
> the unhashed values that must match.
>
> Typos:
>
> P12: ', parameters' -> ', or parameter'
> P15: 'means' -> 'mean'
> P17: 'Breadh' -> 'Breadth'
> P19: 'beleif' -> 'belief'

_______________________________________________
secdir mailing list
[email-masked]
https://mailman.mit.edu/mailman/listinfo/secdir
_______________________________________________
secdir mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/secdir

---

From: Cullen Jennings <[email-masked]>
Date: Sun, 19 Oct 2008 12:01:57 -0600

On Oct 17, 2008, at 5:41 PM, DRAGE, Keith (Keith) wrote:

> There were proposals a while back to decrease the recommended  
> initial value of the Max-Forwards header from 70 to 20, but they do  
> not seem to have come to anything. Is this something that we believe  
> should now happen?

My recollection was that when this got discussed in the WG, there were  
a lot of concerns about how much one could reduce it, and even if it  
was moved to 20, it does not really help much in any practical sense.  
I would not want to do this sort of change without checking with the WG.

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Sat, 18 Oct 2008 01:41:04 +0200

I read:

> partial solutions that improve the situation - perhaps be enough, but 
> I can't be sure. These include:

So to a certain extent assumed that what followed were not necessarily expected to be followed through. 

For:

> There should be a real time bound on any forwarded SIP request, after 
> which the request is deleted and its state removed. I don't know the 
> appropriate real time bound, but I can't imagine any reason to let a 
> request bounce around in the network for more than a week.

I assumed that because of the provided mechanisms we would avoid system overload due to multiple parallel forking, and that therefore processing power would let us get to the expiry of the Max-Forwards header within a reasonable time. 

There were proposals a while back to decrease the recommended initial value of the Max-Forwards header from 70 to 20, but they do not seem to have come to anything. Is this something that we believe should now happen?

Regards

Keith

> -----Original Message-----
> From: Cullen Jennings [mailto:[email-masked]] 
> Sent: Friday, October 17, 2008 8:26 PM
> To: Charlie Kaufman
> Cc: [email-masked]; IESG IESG; Robert Sparks; 
> [email-masked]; [email-masked]; Byron 
> Campen; Dean Willis; DRAGE, Keith (Keith); Dan Romascanu
> Subject: Re: Secdir review of draft-ietf-sip-fork-loop-fix-07.txt
> 
> 
> Hi All,
> 
> I have put this draft on the agenda for next weeks IESG call 
> but I have not seen much discussion of what if any changes 
> should be made given the review below. The unfortunate 
> problem is we don't have any particularly good solutions for 
> the loop issue and mostly need to decide what we can get 
> published and implemented that is significantly better than 3261.
> 
> Cullen
> 
> > security area directors.  Document editors and WG chairs 
> should treat 
> > these comments just like any other last call comments. Feel free to 
> > forward to any appropriate forum.
> >
> > accidental - misconfiguration, the document has been 
> revised so as to 
> > make such loops somewhat less likely to occur by accident 
> and somewhat 
> > less disastrous for network loading when they do occur. The attacks 
> > and their consequences are also much better explained in 
> the document. 
> > Unfortunately, they don't solve the fundamental problem of having a 
> > single request generating an essentially infinite load. It 
> still seems 
> > mailing lists each contain the other as a member, any 
> message sent to 
> > either list ends up cycling between the two list processors sending 
> > continuous streams of messages to other members of both 
> lists. In the 
> > Internet's younger days, that happened routinely. If three Internet 
> > mailing lists each contain the other two lists as members, 
> and if the 
> > expanders are smart enough to send out copies of the message in 
> > parallel to the multiple recipients, then not only do other members 
> > get a continuous stream of messages but the list expanders will be 
> > overwhelmed by all of the parallel requests. I don't know 
> exactly what 
> > solutions we have found to this problem, but it's possible 
> it could be 
> > retrofit onto SIP.
> >
> 'presence' 
> > of someone or something so as to be able to direct an 
> instant message, 
> > phone call, or some such. Each place where it looks for someone may 
> > (like a mailing list) forward the query onto a number of other SIP 
> > servers. SIP does detect loops, but in remains the case 
> that if there 
> > are N places to look for someone, and each of those N 
> places points to 
> > the other N-1 places (a not unlikely naïve configuration), 
> there will 
> > be N! queries processed before the search terminates. For N of even 
> > moderate size, that can effectively be an infinite loop.
> >
> > the spec is that it limits the parallelism of the 
> processing of these 
> > N! queries so that a single bad query cannot deny service to other 
> > users of the SIP server. With the sample "Max-Breadth" of 60, it 
> > probably does not take a very large number of these 
> infinite loops to 
> > overwhelm the system.
> >
> > of queries in the processing of a single request would 
> break too many 
> > real-world scenarios. Other possibilities were discussed on the SIP 
> > mailing list and this was apparently the consensus approach.
> >
> > partial solutions that improve the situation - perhaps be 
> enough, but 
> > I can't be sure. These include:
> >
> > examine. This could be done by suspending the identity 
> being searched 
> > for, or if enough information is present only suspending the 
> > forwarding address that generated the loop. The upside of 
> this scheme 
> > is that loops will be broken quickly. The downside is that 
> it creates 
> > a new security exposure where someone intentionally 'fakes'
> > There should be a real time bound on any forwarded SIP 
> request, after 
> > which the request is deleted and its state removed. I don't 
> know the 
> > appropriate real time bound, but I can't imagine any reason 
> to let a 
> > request bounce around in the network for more than a week.
> > should do in the case of overload. If particular, crashing 
> is probably 
> > not a good idea; overload should be expected. If the table of 
> > outstanding forwarded requests overflows (as it will in 
> error and some 
> > non-error cases), there is a trade-off between discarding 
> the oldest 
> > request, the newest request, the most forwarded request, the least 
> > forwarded request, etc. When the capacity for incoming 
> connections is 
> > exceeded, it may be better to close some existing ones to allow new 
> > ones to arrive.
> >
> > cheaper by using a hash function over the repeating data 
> and comparing 
> > the hashes. The original document suggested MD5. Even 
> though for the 
> > purpose of this document MD5 would be perfectly reasonable, MD5 has 
> > become politically incorrect so the document now recommends 
> CRC-32c as 
> > an alternative. There are two problems with that. One is 
> that 32 bits 
> > are arguably not enough. Failing 1 in 4 billion requests is 
> not very 
> > many, but it will happen. As the forwarding lists get longer (and 
> > particularly if a check is made against outstanding requests) the 
> > failure rate will get much larger.
> > More subtly, CRC-32c is 'linear', so if there is a 
> collision causing a 
> > failure, it may be a solid failure where retries also fail. 
> A better 
> > solution would be to propose a politically correct hash (like
> > hint - there is no requirement that different 
> implementations do this 
> > the same way. The best solution here is to compute a 
> relatively small 
> > hash as a performance hack, but when there is a match check the 
> > unhashed values that must match.
> >
> > Typos:
> >

---

From: Robert Sparks <[email-masked]>
Date: Mon, 20 Oct 2008 13:13:14 -0500

To reinforce Cullen's point here:

This draft makes a really bad situation significantly better - its not  
perfect, and the WG has spend time discussing the kinds of issues  
Charlie points to (I'll respond directly to Charlie's message in a  
moment).

I've seen several implementations of the loop-detection part of this  
draft (and a couple of the max-breadth) already - we retested the  
original doom-scenario at the SIPit last week, and, as anticipated, it  
resulted in quick loop-detection instead of message explosion.

It's important to get this published as a "make it better" step - we  
need to encourage existing deployed implementations to make these  
changes if they haven't already. We can continue work on making it  
even better, but the discussions so far indicate that steps past  
what's here will involve major architectural changes to the core  
protocol.

RjS

On Oct 17, 2008, at 2:25 PM, Cullen Jennings wrote:

> decide what we can get published and implemented that is  
> significantly better than 3261.
>
> Cullen
>
>> somewhat less disastrous for network loading when they do occur.  
>> The attacks and their consequences are also much better explained  
>> in the document. Unfortunately, they don't solve the fundamental  
>> problem of having a single request generating an essentially  
>> infinite load. It still seems unacceptably risky to deploy.
>>
>> members, and if the expanders are smart enough to send out copies  
>> of the message in parallel to the multiple recipients, then not  
>> only do other members get a continuous stream of messages but the  
>> list expanders will be overwhelmed by all of the parallel requests.  
>> I don't know exactly what solutions we have found to this problem,  
>> but it's possible it could be retrofit onto SIP.
>>
>> instant message, phone call, or some such. Each place where it  
>> looks for someone may (like a mailing list) forward the query onto  
>> a number of other SIP servers. SIP does detect loops, but in  
>> remains the case that if there are N places to look for someone,  
>> and each of those N places points to the other N-1 places (a not  
>> unlikely naïve configuration), there will be N! queries processed  
>> before the search terminates. For N of even moderate size, that can  
>> effectively be an infinite loop.
>>
>> The proposal I made in my last review which limited the total  
>> number of queries in the processing of a single request would break  
>> too many real-world scenarios. Other possibilities were discussed  
>> on the SIP mailing list and this was apparently the consensus  
>> approach.
>>
>> When checking for loops, test not only against earlier points in  
>> the history of this query thread, but also against any parallel  
>> queries that are currently waiting for responses from other SIP  
>> servers. This would more quickly catch the overload problem because  
>> in those circumstances the tables of delegated queries would  
>> quickly explode.
>>
>> scheme is that loops will be broken quickly. The downside is that  
>> it creates a new security exposure where someone intentionally  
>> 'fakes' a loop in order to make some entity unfindable.
>>
>> know the appropriate real time bound, but I can't imagine any  
>> reason to let a request bounce around in the network for more than  
>> a week. One example query in the document (section 3) was estimated  
>> to take 3 trillion years to complete.
>>
>> Smaller issues:
>>
>> of outstanding forwarded requests overflows (as it will in error  
>> and some non-error cases), there is a trade-off between discarding  
>> the oldest request, the newest request, the most forwarded request,  
>> the least forwarded request, etc. When the capacity for incoming  
>> reasonable, MD5 has become politically incorrect so the document  
>> now recommends CRC-32c as an alternative. There are two problems  
>> with that. One is that 32 bits are arguably not enough. Failing 1  
>> in 4 billion requests is not very many, but it will happen. As the  
>> against outstanding requests) the failure rate will get much  
>> larger. More subtly, CRC-32c is 'linear', so if there is a  
>> collision causing a failure, it may be a solid failure where  
>> retries also fail. A better solution would be to propose a  
>> politically correct hash (like 64 bits of a SHA-256) and put a hint/ 
>> nudge that MD5 (or even MD4) would be just fine and explain why.  
>> This is only an implementation hint - there is no requirement that  
>> different implementations do this the same way. The best solution  
>> here is to compute a relatively small hash as a performance hack,  
>> but when there is a match check the unhashed values that must match.
>>
>> Typos:
>>

---

From: Robert Sparks <[email-masked]>
Date: Mon, 20 Oct 2008 13:52:07 -0500

Hi Charlie -

Apologies for the delayed response - I saw this right before heading  
to SIPit last week (and last week was really busy).

Some comments inline:

> still seems unacceptably risky to deploy.

We beat on exactly this question in the WG when we were discussing  
this revision. The group recognizes that there is still a path through  
this can stimulate a very large number of message, even if they are  
spread over a long period of time. The consensus was that this is a  
significant enough improvement to move forward with it. To completely  
solve the problem you're describing will likely require fundamental  
protocol changes - things less likely to be quickly implemented and  
deployed than what this draft is describing.

> terminates. For N of even moderate size, that can effectively be an  

(At the risk of opening a different can of worms):

And this is even worse in real deployments right now because there are  
a lot of non-standard things in the network that are not SIP proxies  
but act mostly like intermediaries (things often called B2BUAs or  
SBCs). They _remove_ the historic information as they propagate the  
message - rendering what's in _this_ draft ineffective.

> but I can't be sure. These include:

I see these next two suggestions as reasonable operation/ 
implementation recommendations rather than protocol changes - maybe we  
could add them to the "advice-to" section?

> 3 trillion years to complete.

Allowing a SIP request to pend forever was an intentional design  
choice made very early in the protocol's lifetime - allowing the  
rendezvous function to hunt for new targets forever is a direct, if  
possible unintended, consequence. We have built in mechanisms to both  
endpoints and proxies that allow them to make this hunt (or direct  
pending request) stop. When we did that, particularly for proxies, we  
argued for a _very_ long time about what a good recommended bound  
would be and couldn't develop a good consensus - 3261's discussion of  
Timer C says "must be at least 3 minutes". Essentially, this interacts  
directly with how long you can let a request to start communicating  
remain outstanding - (the early application was "letting a phone  
ring"). And we have real-world scenarios, such as interacting with  
IVRs from really big companies, where phone-calls stay in the ringing  
state the whole time you're interacting with the robot - something  
that can take 10s or 30s of minutes.

> ones to allow new ones to arrive.

There is a separate effort in the WG right now addressing overload.

> the unhashed values that must match.

Well, as you noted, this is only a suggestion in an "advice to  
implementers" section. RFC2543/3261 recommended using MD5, and when  
this went through the working group there was a lot of discussion  
about not needing a cryptographic hash and that speed of computation  
was important. Cullen can comment more perhaps. If I understand your  
point correctly, we anticipated the repeated collision on retry in  
that same section (4.2.4), leading to the recommendation to include  
"at least one field in the input to the hash that varies between  
different transactions".

> P19: 'beleif' -> 'belief'

Thanks for catching those!

RjS