Recent problem records: UrlEncode, UrlDecode

Knowledge to understand before reading this article: When to Use UrlEncode and UrlDecode Functions

The author uses Google Chrome, and analyzes third-party website HTTP protocol interfaces by pressing F12 to capture packets.

Scenario

An operations engineer occasionally uses an outsourcing company's website system to perform device entry tasks. The process is simple:

1. Enter basic device information, about 7 or 8 fields need to be filled in, then click the save button;
2. After the basic information is saved successfully, proceed to select the device type, then click the "Generate Device Identifier" button;
3. After the device identifier is generated successfully, enter the module information associated with the device. Simple devices only need to enter 2 modules, complex devices have 6 modules, each module has 3 or 4 fields to fill in, and finally click save.

Successfully entering one device might take only a few minutes even with the fastest hands, which is not a big deal.

Suddenly the boss says there are 1000 devices to enter? The operations engineer cries 😂. That's when the developer steps in:

1. The operations engineer prepares an Excel template, entering the basic device information and device type information for the 1000 devices that need to be entered. This workload is not large, just half a day, at most one day's work;
2. The developer creates a C/S client tool, configuring the module entry rules in the program according to business requirements;
3. During program execution, each time a device is entered, the generated device identifier is associated with the device;
4. After all entries are completed, provide an Excel export that can export all device basic information and generated device identifiers together. Job done.

After several days of development, the developer hands over the polished tool to the operations engineer, who uses it and gives a look of approval...

Problem

The previous setup is a bit verbose. While developing this tool, the developer encountered a problem:

xxx interface

This is information about a certain interface. Content-Type is application/x-www-form-urlencoded, and the parameters below use Form Data, meaning the parameters are UrlEncoded. For example, a parameter before encoding:

"Content":"{"AP_Name":"HK_7889","IP":"192.168.0.1"}"

After encoding (you can verify using this online URL encoding/decoding tool):

"Content":"%7B%22AP_Name%22%3A%22HK_7889%22%2C%22IP%22%3A%2292.168.0.1%22%7D"

When testing with Postman, without UrlEncoding the parameter, the interface test succeeded. While developing this tool, three interfaces were similar and did not use UrlEncode:

var client = new RestClient("http://admin.lqclass.com/api/device");
client.Timeout = -1;
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/x-www-form-urlencoded");
request.AddParameter("Content", "{\"AP_Name\":\"HK_7889\",\"IP\":\"92.168.0.1\"}");
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);

But when encountering a slightly more complex interface, such as the parameter shown in the screenshot:

"Content":"{"AP_Name":"HK_7889","IP":"192.168.0.1","Module":[{"M_Name":"cameri0","Desc":"cameri0","AP_PUID":"54632325461320320"},{"M_Name":"cameri1","Desc":"cameri1","AP_PUID":"54636325461320320"},{"M_Name":"cameri2","Desc":"cameri2","AP_PUID":"54632325421320320"}]}"

The Content value is clearer when formatted. Module is the module information associated with the device:

{
  "AP_Name": "HK_7889",
  "IP": "192.168.0.1",
  "Module": [
    {
      "M_Name": "cameri0",
      "Desc": "cameri0",
      "AP_PUID": "54632325461320320"
    },
    {
      "M_Name": "cameri1",
      "Desc": "cameri1",
      "AP_PUID": "54636325461320320"
    },
    {
      "M_Name": "cameri2",
      "Desc": "cameri2",
      "AP_PUID": "54632325421320320"
    }
  ]
}

The actual UrlEncoded parameter is:

"Content":"%7B%22AP_Name%22%3A%22HK_7889%22%2C%22IP%22%3A%22192.168.0.1%22%2C%22Module%22%3A%22%255B%257B%2522M_Name%2522%253A%2522cameri0%2522%252C%2522Desc%2522%253A%2522cameri0%2522%252C%2522AP_PUID%2522%253A%252254632325461320320%2522%257D%252C%257B%2522M_Name%2522%253A%2522cameri1%2522%252C%2522Desc%2522%253A%2522cameri1%2522%252C%2522AP_PUID%2522%253A%252254636325461320320%2522%257D%252C%257B%2522M_Name%2522%253A%2522cameri2%2522%252C%2522Desc%2522%253A%2522cameri2%2522%252C%2522AP_PUID%2522%253A%252254632325421320320%2522%257D%255D%22%7D"

Normally, for generic interfaces like the one that succeeded earlier (directly calling without UrlEncode), it works fine.

But for this interface call, the server returned an error message: "xxx parsing failed". The call code is as follows:

var client = new RestClient("http://admin.lqclass.com/api/device");
client.Timeout = -1;
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/x-www-form-urlencoded");
request.AddParameter("Content", "{\"AP_Name\":\"HK_7889\",\"IP\":\"192.168.0.1\",\"Module\":[{\"M_Name\":\"cameri0\",\"Desc\":\"cameri0\",\"AP_PUID\":\"54632325461320320\"},{\"M_Name\":\"cameri1\",\"Desc\":\"cameri1\",\"AP_PUID\":\"54636325461320320\"},{\"M_Name\":\"cameri2\",\"Desc\":\"cameri2\",\"AP_PUID\":\"54632325421320320\"}]}");
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);

What is the difference between the two call code snippets? Only the Content value differs. Finally, I wondered if a manual UrlEncode was needed? It's not a URL parameter, why would encoding be required? Never mind, let's encode it anyway.

Solution

After encoding the parameter, the call:

var client = new RestClient("http://admin.lqclass.com/api/device");
client.Timeout = -1;
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/x-www-form-urlencoded");
request.AddParameter("Content", "%7B%22AP_Name%22%3A%22HK_7889%22%2C%22IP%22%3A%22192.168.0.1%22%2C%22Module%22%3A%22%255B%257B%2522M_Name%2522%253A%2522cameri0%2522%252C%2522Desc%2522%253A%2522cameri0%2522%252C%2522AP_PUID%2522%253A%252254632325461320320%2522%257D%252C%257B%2522M_Name%2522%253A%2522cameri1%2522%252C%2522Desc%2522%253A%2522cameri1%2522%252C%2522AP_PUID%2522%253A%252254636325461320320%2522%257D%252C%257B%2522M_Name%2522%253A%2522cameri2%2522%252C%2522Desc%2522%253A%2522cameri2%2522%252C%2522AP_PUID%2522%253A%252254632325421320320%2522%257D%255D%22%7D");
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);

Haha, it succeeded. Here's a simple guess: the third-party service probably performs a UrlDecode operation on the received parameters.

Actually, there was an intermediate UrlEncode operation on the parameter, specifically on the Module value:

"Content":{"AP_Name":"HK_7889","IP":"192.168.0.1","Module":[{"M_Name":"cameri0","Desc":"cameri0","AP_PUID":"54632325461320320"},{"M_Name":"cameri1","Desc":"cameri1","AP_PUID":"54636325461320320"},{"M_Name":"cameri2","Desc":"cameri2","AP_PUID":"54632325421320320"}]}

First UrlEncode, i.e., first UrlEncode the value of Module:

"Content"：{"AP_Name":"HK_7889","IP":"192.168.0.1","Module":%5B%7B%22M_Name%22%3A%22cameri0%22%2C%22Desc%22%3A%22cameri0%22%2C%22AP_PUID%22%3A%2254632325461320320%22%7D%2C%7B%22M_Name%22%3A%22cameri1%22%2C%22Desc%22%3A%22cameri1%22%2C%22AP_PUID%22%3A%2254636325461320320%22%7D%2C%7B%22M_Name%22%3A%22cameri2%22%2C%22Desc%22%3A%22cameri2%22%2C%22AP_PUID%22%3A%2254632325421320320%22%7D%5D}

The second UrlEncode is the successful parameter method above, where the entire Content value is UrlEncoded. See the successful parameters above, no need to repeat them.

Final Summary

When capturing other people's packets, don't rely solely on impressions or existing knowledge to decide what to do. For example, with the earlier parameters, the call succeeded without UrlEncode. Does that mean I should follow the same approach for other packets? When stuck, try more speculative methods.

Summary: "Just do it, and stop overthinking."

The UrlEncode C# code used in this article:

public static string UrlEncode(string str)
{
    StringBuilder sb = new StringBuilder();
    byte[] byStr = System.Text.Encoding.UTF8.GetBytes(str); // Default is System.Text.Encoding.Default.GetBytes(str)
    for (int i = 0; i < byStr.Length; i++)
    {
        sb.Append(@"%" + Convert.ToString(byStr[i], 16));
    }

    return (sb.ToString());
}