twilio’s inconsistent API

Share/Bookmark

I just spent quite some time debugging an issue in my Rails application where it would not send a text message to a certain client and instead crash with an "invalid phone number" message. It worked with all the other clients, but not with this one. 

After adding dozens of lines of "logger.debug" to the code, I finally realized the problem: This client had the phone number stored in the format "49130544664", while all the other clients used the format "+49130544664". 

This is what it came down to: the international prefix was not stored in the database for this entry. 
This client has been in the database for MONTHS and everything worked fine when I CALLED this client. The twilio API didn't bother the least bit whether or not the prefix was present. 
 

This code works like a charm (if you replace ACCOUNT_ID, AUTH_TOKEN and phone numbers with reasonable values): 

twc = Twilio::REST::Client.new ACCOUNT_ID, AUTH_TOKEN
twc.account.calls.create({ to: '4984654646', from: '+1454545454545', url: 'http://www.hiddenfeatures.net'})

 

However, trying to send a TEXT MESSAGE with the following code will crash your application: 

twc = Twilio::REST::Client.new ACCOUNT_ID, AUTH_TOKEN
twc.account.sms.messages.create({ to: '4984654646', from: '+1454545454545', body: 'http://www.hiddenfeatures.net'})

 

And all you need to do, to make this working again is replace to: '4984654646' with to: '+4984654646'

That's it: Sending an SMS requires the international prefix, making a call does not.

Programming quotes

This is just a wild list of programming quotes I found on the InternetZ. 

It is not supposed to be educational or anything of any value. Just for my own entertainment.

 

 

"It's not that I'm surrounded by incompetence that bothers me, it's that I fit in so well."

"I don't care if it works on your machine! We are not shipping your machine!"    — Vidiu Platon

"Assumption is the mother of all fuckups."    — Anonymous.

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."    — Brian W. Kernighan.

"The best thing about a boolean is even if you are wrong, you are only off by a bit."

"Evolution is God's way of issuing upgrades."

"It should be noted that no ethically-trained software engineer would ever consent to write a DestroyBaghdad procedure. Basic professional ethics would instead require him to write a DestroyCity procedure, to which Baghdad could be given as a parameter. "    — Nathaniel Borenstein.

"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live."    — Martin Golding.

"Programming is like sex: one mistake and you're providing support for a lifetime."    — Michael Sinz.

Sources include: 

Recent change to the twilio API breaks existing code

Apparently twilio recently made some changes to the undocumented parts of their API. Something that software developers and programmers that rely on this behaviour constantly dread. 

The change that broke my application, was relying on the fact, that twilio would pass a parameter "PhoneNumber" in the asynchronous callback after verification of an outgoing caller id.

So for the last 4-5 months the following code worked perfectly well:

 if("success" == params[:VerificationStatus]) @customer_initialization.verified_number(@customer_initialization.customer, params[:PhoneNumber]) 

 

Suddenly, the code didn't work any more. So I dug through my code, trying to debug the issue and finally found the culprit: twilio was no longer passing along params[:PhoneNumber]. I immediately went to check the documentation and suddenly realized, that I misread it. Officially params[:PhoneNumber] was never part of the API. The phone number is only documented for the response to the original API call – not for the status callback. 

The only official way is to retrieve params[:OutgoingCallerIdSid] from the callback and query the twilio API with that SID. Which of course adds complexity and worst of all latency to your application. Fortunately, for the time being twilio also passes params[:To] and params[:Called] which both contain the desired phone number in e.164 format. But again: Those fields are not documented – so use at your own risk. 

 phone_number = par[:To] || twilioclient.account.outgoing_caller_ids.get(ociSID).phone_number 

 

This is the code I now use – feel free to copy it

 

My thoughts on these kind of changes

I am a big fan of backwards compatibility and in my humble opinion much of the success of Microsoft can be attributed to their outstanding efforts in ensuring backwards compatibility between different versions of their Windows operating system. I think that all software developers and programmers can learn a few things from Microsoft in this regard – no matter what your feelings are towards this company. 
One of the best sources on this topic is Raymond Chen's blog The Old New Thing and especially this post here: http://blogs.msdn.com/b/oldnewthing/archive/2003/12/24/45779.aspx

There is also a pretty good post covering the same topic on the Joel on Software blog here: How Microsoft Lost the API War

This whole topic is of course a double edged sword. On the one hand you keep your users happy, because their software just keeps on working with every new version of your operating system (or API). On the other hand you don't adapt to changes in the environment very fast (just think of the fact that almost every user on Windows XP had full administrative privileges and in some way this is still true for Windows Vista / 7 / 8 – despite this behaviour being considered harmful by the majority of software developers and programmer). 
 

However, in most instances users (and software developers using your API) just want their software to freaking work – so you should think long and hard before making breaking changes (another example by Microsoft: It took them 4 versions of their C# programming language to change this error-prone behaviour of the foreach loop)

Finding Source Code Plagiarism

 

I don't like it when people cheat on exams. I don't like people who steal other people's (intellectual) property.
Because of that I did not hesitate for too long, when I was given the opportunity to write a plagiarism detection system as part of my bachelor thesis.

The result of this work is now available on GitHub: https://github.com/yas4891/MUTEX

Background

Software Plagiarism has been rampant among students at my university (http://www.unibw.de) . My analysis in preparation for this project showed that the quota of ripped-off solutions handed in by students was generally around 25 percent and could get as high as 60 percent. Of course, this all depends on the difficulty of the assigned task. The more difficult the task is, the more students will try to cheat.

Plagiarism does not help anyone. The students who cheat don't put in the necessary effort to understand and solve the problem at hand. They subsequently do not learn programming and often fail exams. The professors get annoyed by seeing the same source code over and over again. It is essentially a loose-loose situation.

 

The Idea

To keep students away from software plagiarism and to improve the level of coding competency among students, it was decided to create a self-written IDE to be used by the students. The feature list includes  functional and non-functional tests as well as plagiarism detection. The idea is to automatize the procedure of handing in coding assignments as much as reasonably possible, while at the same time making it harder for students to cheat the system. The new system should rigorously enforce software development best practices (through both functional and non-functional testing) and force every student to hand in his very own solution to the problem.

The student will use the IDE to develop their solution. When she thinks that she's solved the assigned task, she hits a button in the IDE. The source code is then send to a central server, where it is tested and checked for plagiarism. The plagiarism detection is done against the set of previously gathered source codes for the same task. This means that the first student to hand in a correct solution will automatically be judged as having presented a unique source code. The second student's solution is tested against the first solution. The third solution against the second and first solutions and so on.

If the source code does not pass both the tests and plagiarism detection, the student has an unlimited number of retries to rewrite the program.

My job in this over-arching project was to implement a system to reliably detect software plagiarism. Because of the iterative nature of the process layed out for the students – very similar to the way Test Driven Development is performed – the plagiarism detection system has to be fast enough to support this approach.

In absence of a better idea, the plagiarism detection system was named "MUTEX (Munich University Theft EXclusion)".

Requirements

The following requirements were layed out for the plagiarism detection system:

  • comparison of a new source code against a set of 20 source codes should take no longer than 10 seconds
  • programming language of the source codes: C (with an eye on possible extensions for C++, Java and VHDL)
  • runnable as a CGI script withhin a custom written webserver
  • operating system: Windows 7 x64

 

Detecting source code plagiarism

Now, lets have a look at how you actually find matching parts between two source files. The implementation language for this project is C# with .NET Framework 4.0 – in case you were wondering.

The problem of building a plagiarism detection system can be broken down into roughly two chunks – parsing the source code into tokens and comparing token sequences for matching sub sequences.
In this post I will focus solely on finding matching sub sequences shared between two token streams.

 

Definitions

I will use words and notations, that need some more explanation. So here it goes:
 

sequence an ordered list of elements – e.g. strings or token streams. Notation: [h,e,l,l,o] or "hello"
subsequence a part of a sequence
Token a single element of a sequence. Tokens can be marked as already matched
Match an identical subsequence that is present in both sequences. Matches are not permanent and only existing for the length of one iteration of the algorithm
Tile a permanent and unique match. After the algorithm has finished only tiles and unmarked portions of the sequences remain
MinimumMatchLength / MML configurable parameter of the Greedy-String-Tiling-algorithm. Defines the minimum length of a match to be considered by the algorithm (and when calculating the similarity)
similarity the ratio of identical tokens between two sequences given in percent.
similarity = (sum(length of all matches) / length of shorter sequence)
maximum similarity given a new sequence NEW and a set of old sequences OLD_SET: the highest calculated similarity between NEW and any one element of OLD_SET
local confusion describes an alteration to the token stream that causes the algorithm to miss a otherwise valid match. Inserting a big enough number of locally confusing alterations into a token stream will make it look like valid, unique source code to the algorithm

 

Greedy-String-Tiling

The Greedy-String-Tiling algorithm was originally devised by Michael J. Wise at the University of Sydney in 1993. If you like reading the works of people that easily outsmart you as much as I do, read his original report here.

The algorithm works on two sequences A and B. It consists of a total of five loops and can be broken into two phases. Phase 1 is used to find the longest matches in the remaining, yet unmarked portions of the two sequences; phase 2 marks the tokens of the longest matches found, thereby converting the match into a tile. Those two phases are repeated over and over again until no match longer than the MML is found. 

Before Phase 1 we set LastMatchLength to MinimumMatchLength(MML) – so that if no longer match is found during this iteration the algorithm will terminate – and create a list to store the matches found in this iteration. In Phase 1 we iterate over all unmarked tokens in A and compare it with all the unmarked tokens in B. When it finds a match, it uses a third loop to extend the match as far possible (hence "greedy"). The loop takes the subsequent tokens in A and B as long as they match and are  unmarked. After extending the match to its maximum, the algorithm checks whether the match is longer than or of equal length as the matches previously found during this iteration. If it is longer than the previous matches the list is reset and the new match is added as the only element into this list. If the match is of equal length it is simple appended to the list.
After phase 1 of each iteration we have a list that contains the matches with maximum length in the remaining parts of the sequence. With each iteration the length of the matches found decreases, eventually terminating the algorithm.

In phase 2 we iterate over all found matches, mark all the tokens in each match as used and store the match in the tiles collection (promoting the match to tile).
The relevant code for this simple GST implementation can be found in GSTLibrary.tile.GSTAlgorithm.

 ///</p>
<p>&nbsp;</p>
<p><summary> /// performs exactly one iteration of the algorithm /// </summary> public override void DoOneRun() { if(Finished) throw new GSTException(&quot;algorithm is finished&quot;); var watch = Stopwatch.StartNew(); var listMatches = new List<tile<t>&gt;(); LastMaximumMatch = MinimumMatchLength; // for every token in A that is unmarked foreach(var tA in ListA.Where(t =&gt; !t.Marked)) { var tokA = tA; // CLOSURE int indA = ListA.IndexOf(tokA); // for every token in B that is unmarked and matches tokA foreach(var tB in ListB.Where(t =&gt; !t.Marked).Where(tokA.EqualsTokenValue)) { //counter++; var tokB = tB; // CLOSURE int indB = ListB.IndexOf(tokB); int matchLength = CalculateMatchLength(indA, indB); if(matchLength &gt;= LastMaximumMatch) { var tile = AbstractGSTAlgorithm.CreateTile(ListA, indA, indB, matchLength); AddTileToMatchList(listMatches, matchLength, tile); } } // foreach in B } // foreach in A foreach (var tile in listMatches) { MarkTileAsMatched(tile); ListTiles.Add(tile); } TilesMatchedInLastRun = listMatches; //Console.WriteLine(&quot;one run({1}) took {0} ms&quot;, watch.ElapsedMilliseconds, counter); } 

 

 

runtime complexity of the Greedy-String-Tiling algorithm

The complexity of the algorithm is mainly determined by phase 1 and its three loops. The worst-case complexity is O(n3), best-case complexity is O(n2) – where n is the sequence length of the shorter sequence. The experienced complexity tends to be near to O(n3).

From my experience you can expect this simple implementation of the Greedy-String-Tiling algorithm to exceed runtimes of one second (exactly 1596 ms) at sequence lengths of about 900 tokens. With 1700 tokens you will see runtimes of about 11 seconds (11395 ms) on modern hardware (1.88X tokens, 7.3X runtime compared to 900 tokens). At about 2900 tokens the runtime will exceed 60 seconds (63909 ms; 3.2X tokens, 40.04X runtime).

As most of the source codes handed in by the students tend to be shorter than 900 tokens, the performance seems to be reasonable for our use case.
 

Conclusion

The Greedy-String-Tiling algorithm is a great heuristic to find similarities shared between two sequences. The implementation shown here is not optimized in any way, although there exists an optimization that reduces the average complexity to near O(n). This optimization will be discussed in one of the upcoming posts on this topic. The series will continue with posts on how to hide plagiarism and the creation of a special grammar for MUTEX to improve the detection rate.

Twilio Demo Application and Slides

About 2 weeks ago (tempus fugit) I did a short presentation of the twilio phone communication API at the local Ruby User Group Meetup. 

I promised to share those, but didn't find the time to do so. 
So without further ado, here they are:

https://github.com/yas4891/twilio-demo (the demo application – written with Ruby on Rails)

Twilio Demo Slides

 

I hope to update this post in the future with some more information, but I've got a bit too much on my plate right now.

 

 

 

Announcing Talk at Rubyshift Munich

Hi folks, 

just a quick heads up. I will talk about the twilio API (http://twilio.com) at Rubyshift Munich Meetup on 08.08.2012 at 19:30 (7:30 pm).

The meetup will take place at :
coworkingmunich.de
Sonnenstraße 32
80331 München

Topics covered :

- buying a phone number

- creating and setting up the twilio application (model on their part – not the coding)

- pricing

- restrictions and caveats

 

I have also prepared a demo application (code based on Ruby on Rails)

The Demo App covers: (all via API)

- verifying a phone number you own

- send / receive text messages

- call a phone 

- read messages via speech engine

- playback mp3 

- gather key press

- make a conference call

- record a call

- transcribe a recording

 

 

I hope to see you there 

why programmers suck less at life

That's one hell of a title – but I'm an attention whore, so you all got to live with it :-)

First of all, I'd like to thank Benedikt Deicke (http://www.synatic.net/), who inspired me for this blog post. He's a terrific software developer and generally a great guy. 

 

Your experiences change your look at the world at large. So here is a list of reasons why being a programmer helps you succeed in life.

 

1. It is (almost) always your fault

This is arguably one of the most important lessons to succeed in life – it certainly is the most important lesson for programmers. On an all too regular basis I see developers sitting in front of the computer, complaining about how their application is not working – although they have written perfect code! Probably something wrong with the compiler, the CPU, the OS or increased solar activity.
Wrong, WRONG, SOOOO WRONG. Whenever something is not working out in your application, it is your fault. The probability of the problem not being with your code is almost zero. It is a hard lesson to learn. It is hard on your ego, but the sooner you swallow it, the sooner you can go and find a solution to your problem. 
But this is also a good thing: If it is your fault, you can do something about it!

That is also true for real life. Most of the things you are whining about, are based on decisions YOU made in the past. 

  • Getting a D in your latest exam? Probably didn't start learning until 5 days before the exam!
  • Not getting the pay raise you deserve? Take an honest look, chances are the people around you are working harder and are getting things done! (Or: You didn't bargain hard enough!)
  • Not satisfied with your job? You haven't sent an application for another job either!

 

This list goes on and on! Take matters into your own hands and start working on the things that frustrate you! Change something TODAY!

 

2. Optimize All The Things!

Benedikt pointed this habit out to me. We both tend to optimize workflows and systems to their fullest extend. We measure, learn, optimize on a daily basis.
We both love to go into our code, refactor it, make it more readable, increase performance and extensibility. 
And we both show this same behaviour in our everyday life – sometimes to the dismay of our spouses.

When driving to work I know the interval of all the traffic lights along the route. I know exactly when it is useful to accelerate and when not to. I know when it is fuel efficient to turn off the engine at a red light, just by observing the traffic around me.
The results? A 20 % decrease in fuel consumption compared to my first month driving this same route – and hundreds of Euros saved in fuel costs each year.

Benedikt even found the best way to cross roads on the way to work – as a pedestrian and depending on time of day.

 

 

inconsistencies when accessing the registry on Windows x64

ABSTRACT

This article shows a problem that exists when trying to obtain the path of Environment.SpecialFolder.CommonDocuments from a x86 process on systems running Windows x64 after the Common Documents folder has been moved from the default location. The article explains how to reproduce the problem and shows some working code to work around this bug. Additionally this article explains the background behind this problem and some implementation details of x86 emulation on Windows x64 systems.

 

Background

The 64-bit (x64) versions of Windows supports 32-bit (x86) processes through an emulation concept called 'Windows on Windows 64' (WOW64). Part of the magic are the File System Redirector and the Registry Redirector. The File System Redirector comes into play, when you try to access certain folders- e.g. %windir%\system32 – from an x86 process. It will map the request transparently to %windir%\SysWOW64. Registry Redirector does exactthe same for certain nodes in the registry. For example, HKEY_LOCAL_MACHINE\Software is redirected to HKEY_LOCAL_MACHINE\Software\Wow6432Node. 

Additionally, Windows allows users to move those special folders. For example, a user may decide to move his Public Documents folder to a network share, from which backups are created by the local administrator.

Here are the steps you need to take to move the Public Documents folder: 
Or use this alternative way for (moving the Public folders on Windows 7)

  1. Deactivate User Account Control (UAC) in the control panel
  2. Reboot Windows
  3. Navigate to %SYSTEMDRIVE%\Users\Public
  4. Right click on 'Public Documents' (or whatever is the localized name of that folder)
  5. Select 'Properties'
  6. switch to the 'Location' tab
  7. input the new path and apply settings, confirm all questions asked

Note: This is only possibly after deactivating UAC and restarting Windows, so Windows tries to protect you to some extend.

 

The Problem

When a user moves his special folders, all seems well (all IS well) – when viewed from the x64 world. When you look at the folder in Windows Explorer you see the real name (e.g. D:\Real Common Documents) replaced by the localized name ("Public Documents" or "Öffentliche Dokumente" on German versions of Windows).
But once you try to access that same information from any program that was compiled as x86 and runs on the WOW64 emulation, you'll get very different results indeed. 

For demonstration purposes, I moved the Public Documents folder on my Windows 7 x64 computer from its usual place ( C:\Users\Public\Documents ) to D:\ Real Common Documents.

To illustrate I pulled together a very short program in C# and compiled it with .NET 4.0.

namespace GetSpecialFolders
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(Environment.GetFolderPath(Environment.SpecialFolder.CommonDocuments));
        }
    }
}

Then I fired up a command line and ran the following commands:
(Outputs are bold, actual commands are preceded by '>' )

>csc /platform:x86 Program.cs > dev.null

>Program.exe
C:\Users\Public\Documents

>csc /platform:x64 Program.cs > dev.null

>Program.exe
D:\Real Common Documents

(You can also have a look at the screen shot of this experiment)

As you can see the returned values differ between x64 and x86 versions.

 

Explaining the Behaviour

First and Foremost: This behaviour is (somwewhat) by design. The path to the Public Documents folder is stored in the registry and it is done so twice.
registry path for x64: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Common Documents
registry path for x86: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Common Documents

So from my point of view it looks like moving the folder will update the x64 part, but not the x86 part of the registry. 

The following folders are also stored in the 32 bit part of the registry – and are thus likely affected by the same bug: 

  • Common Administrative Tools
  • Common AppData
  • Common Desktop
  • Common Documents
  • Common Programs
  • Common Start Menu
  • Common Startup
  • Common Templates
  • CommonMusic
  • CommonPictures
  • CommonVideo

 

Important notes: 

As you see there are no user-specific folders in this list. Doing the same steps with My Documents folder and testing with a slightly modified version of the above application showed that My Documents is NOT affected by this bug – i.e. the same path is yielded in x86 and x64.
Windows does not allow you to move all of the above folders. At least not with a convenient user interface – you might be able to pull that off through the registry however. 

 

How to solve this problem

The only work around I know of, is the application (and its derivates) I posted above – compiled with /platform:x64. 
Calling the resulting executable and parsing its output solved the problem for me and will likely solve it for you. 

I tried calling SHGetKnownFolderPath directly through P/Invoke while debugging my application, but that unsurprisingly did not help me. 

 

Further Reading:

 

UPDATE 1:

Running the same test (see above) on Windows 8 x64 Consumer Preview returns an empty string for the x86 version. 
So this matter seems to actually get worse with Windows 8!

Survey on the Status Report Framework

I'm currently quite busy with founding a new startup, which is why I post scarcely.
The idea behind my new startup is to lower costs in laboratories by increasing automation.

Expect to read more about this topic as soon as I find the time.

Meanwhile, please take part in a quick 3-5 minute survey on the Status Report Framework and let me hear which features you would like to have implemented.
 

5 tips for non-english software developers / programmers

The inspiration for today's post comes from a piece of code – more precisely from one of those famous // TODO comments.
So here's the comment alongside with the line of code:
 

                // TODO unabhängig von Sprache machen
                properties["authorizedGroup"] = "Benutzer";

You are already wondering what "unabhängig von Sprache machen" and "Benutzer" mean – right?

This is the first thing to avoid:

1. never, NEVER EVER – not even while hacking some website with a gun pointed to your head and a blonde giving you head (Swordfish)… OK maybe this is a valid excuse to not comment at all – write your comments in any other language than English.

You will write 100% of your code in English and if you decide to write your comments in German/Polish/Esperanto/Swahili you'll force your brain to switch context dozens of times while reading your code. There is conclusive evidence that switching contexts disrupts your efficiency while coding.  So just don't do it. Write in English. You'll probably improve your English skills along the way.

 

2. Learn English

Learn to read and write English – proper English. I flinch everytime someone storms into the C# chat over on StackOverflow and writes sentences like this: "I gut prob w/ ASP.NET MVC3. Do u any exp w/ it?"
This sounds like the writer has learned English from the graffitis in the background of an American movie with subtitles. Probably from the movie Dangerous Minds or worse.
Extra tip for readers from India: Putting a 'sir' behind every sentence does improve neither spelling nor grammar.

 

3. Write about programming – in English

I started this blog for exactly that reason: I want to better communicate my ideas and thoughts about programming. I want to be able to do that in English, because English is the lingua franca for software developers. I want to be a great programmer and I share Joel's opinion, that you the difference between tolerable and great is ability to communicate. After graduation from university I am going to be a military leader of 40-200 men and women. I need to be able to communicate in a concise way and yet not miss important details.
If starting your own blog seems like to much trouble for you, go over to StackOverflow and start answering questions. Try to make your answer the most awesome answer on the web.

 

4. Install only English versions of your IDE

This is especially true if your IDE is being shipped by Microsoft. For all that Microsoft does really well with its OS, the programming languages, the framework, world peace (OK maybe not with that), the fact is they… ummm… suck at translations. I know professional interpreters that outright refuse to do any work for Microsoft, because of their translation policies.
The number of times I've been trying to find the Microsoft translation (English to German) of a menu item in Visual Studio is beyond count. Often times this boils down to "third menu from the right, fourth item from top" in the end.

 

5. When logging error messages always use English messages

For the same reasons stated earlier on, I recommend you log all your exceptions / errors in English. This is especially true for exceptions that are automatically translated into the computer's locale – as happens with most exceptions in .NET. Imagine you are receiving a logfile from a customer in Libya and find the error messages written right to left in an unfamiliar alphabet. Good luck finding that bug.
This can be worked around for most exceptions in .NET (solution on StackOverflow) and I'm sure you'll find a way to make it work in your programming language.

 

6. Bonus tip: get the job done

No matter whether you are a native English-speaker or not, the main part of being a software developer is getting things done.

 

You have more tips to share?

Please post them in the comments and keep discussing