Debugging Web Applications with curl: Best Practices and Useful Scripts

By gerald, 3 April, 2023
curl

Photo by Fabricio Trujillo: https://www.pexels.com/photo/grey-laptop-keypad-60626/

When it comes to debugging web applications, one tool that is incredibly useful is curl. Curl is a command-line tool for transferring data between servers and clients. It is especially useful for debugging because it allows you to easily send HTTP requests and view the responses. In this blog post, we will explore some best practices, tips, and tricks for using curl in web application debugging, as well as provide some valuable curl scripts.

Best Practices

  1. Use verbose mode: When debugging with curl, use the verbose mode to see the details of the HTTP requests and responses. Add the -v or --verbose flag to your curl command to enable verbose mode.

  2. Use headers: Use headers to specify any additional information that you need to send with your HTTP request. This can be especially useful for debugging authentication-related issues. Example:

    curl -H "Authorization: Bearer abc123" https://example.com

    In this example, the -H flag is used to add a custom header to the request. The header being added is an Authorization header with a bearer token value of abc123.

    You can add multiple headers to a request by using the -H flag multiple times:

    curl -H "Authorization: Bearer abc123" -H "Content-Type: application/json" https://example.com

    This command will add two headers to the request: an Authorization header with a bearer token value of abc123, and a Content-Type header with a value of application/json.

    Adding custom headers can be useful for authentication, specifying the content type of a request, or providing additional metadata about the request.

  3. Use the -X flag: The -X flag allows you to specify the HTTP method that you want to use. This can be useful if you need to send a request using a different method than the default. Example:

    curl -X POST -d '{"name": "John Doe", "email": "john@example.com"}' https://example.com/api/users

    In this example, the -X flag is used to specify the HTTP method of the request. The method being used is POST, which is used to submit data to the server.

    You can use the -X flag to specify any HTTP method, including GET, POST, PUT, DELETE, and more. For example:

    curl -X PUT -d '{"name": "Jane Doe", "email": "jane@example.com"}' https://example.com/api/users/123

    In this command, the -X flag is used to specify the PUT HTTP method, which is typically used to update an existing resource on the server. The -d flag is used to add a JSON payload to the request.

    Using the -X flag can be useful when you need to specify a non-standard HTTP method, such as when working with a RESTful API.

  4. Use data payloads: If you need to send data with your HTTP request, use the -d flag to specify the data payload. Example:

    curl -d '{"name": "John Doe", "email": "john@example.com"}' https://example.com

    In this example, the -d flag is used to add a JSON payload to the request. The payload being added is a JSON object with two properties: name and email. The contents of the JSON object are enclosed in single quotes and passed as a string to the -d flag.

    You can also pass payloads as key-value pairs using the -d flag. For example:

    curl -d "name=John+Doe&email=john@example.com" https://example.com

    In this command, the -d flag is used to add a payload consisting of two key-value pairs: name=John+Doe and email=john@example.com.

    Adding payloads can be useful for sending data to a server, such as submitting a form or posting data to an API. You can use different formats for payloads, including JSON, URL-encoded data, and raw data. You can specify the content type of the payload using the -H flag. For example:

    curl -H "Content-Type: application/json" -d '{"name": "John Doe", "email": "john@example.com"}' https://example.com

    This command specifies that the payload is in JSON format using the Content-Type header, and adds the JSON payload using the -d flag.

  5. Use cookies: If your application uses cookies for authentication, make sure to include them in your HTTP requests. Example:
    curl -b "sessionid=abc123; csrftoken=xyz456" https://example.com

    In this example, the -b flag is used to add cookies to the request. The cookies being added are sessionid with a value of abc123 and csrftoken with a value of xyz456.

    You can also save the cookies from a previous response and reuse them in a subsequent request using the -c and -b flags. For example:

    curl -c cookies.txt https://example.com/login curl -b cookies.txt https://example.com/dashboard

    In this example, the first command sends a request to the login page and saves the cookies in a file called cookies.txt using the -c flag. The second command reads the cookies from the cookies.txt file using the -b flag and sends a request to the dashboard page.

    Adding cookies can be useful for maintaining session state across requests, as many web applications use cookies to store user session data. You can use the -c flag to save cookies from a response and the -b flag to send cookies in a subsequent request.

Tips and Tricks

  1. Set the user-agent: Some web applications may behave differently based on the user-agent string. To set the user-agent string, use the -A flag followed by the user-agent string. Example:

    curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" https://example.com

    In this example, the -A flag is used to set the user agent string to a specific value. The user agent string in this example is for Google Chrome on a Windows 10 operating system.

    By setting the user agent string to a specific value, you can test how your web application behaves with different user agents. For example, some websites may serve different content or styles based on the user agent string. By changing the user agent string in curl, you can test how your web application behaves for different types of clients.

    Note that some web applications may block requests that have a user agent string that doesn't match a browser or other known client. In such cases, you may need to set the user agent string to a valid value to avoid being blocked.

  2. Use SSL/TLS: If your web application uses SSL/TLS, make sure to use the -k or --insecure flag to allow curl to accept self-signed certificates. Example:

    curl -k https://example.com

    In this example, the -k flag is used to allow insecure SSL connections. By default, curl will verify the SSL certificate of the server and will fail the request if the certificate is invalid. However, sometimes you may encounter situations where the SSL certificate of the server is not valid (for example, if it's self-signed or expired), but you still want to make the request. In such cases, you can use the -k flag to allow the connection to proceed even if the SSL certificate is not valid.

    It's worth noting that using the -k flag is generally not recommended as it can expose you to security risks. If you use -k, you should only use it for testing purposes and not for production use.

  3. Follow redirects: If your application uses redirects, use the -L or --location flag to follow them. Example:

    curl -L https://example.com

    In this example, the -L flag is used to follow redirects. By default, curl will not follow HTTP redirects (such as 301 or 302 responses) and will simply return the redirect response to the client. However, sometimes you may want to follow redirects and retrieve the content of the final URL. In such cases, you can use the -L flag to instruct curl to follow any HTTP redirects.

    It's worth noting that when using the -L flag, curl will only follow up to a certain number of redirects (by default, up to 50). If the number of redirects exceeds the maximum number, curl will return an error. You can customize the maximum number of redirects by using the --max-redirs flag, followed by the number of redirects you want to allow. For example:
    curl -L --max-redirs 10 https://example.com
    This command will follow up to 10 HTTP redirects before returning an error.

  4. Use output redirection: Use output redirection to save the HTTP response to a file. This can be useful for further analysis or sharing with other team members. Example:

    curl https://example.com > output.txt

    In this example, the > symbol is used to redirect the output of curl to a file called output.txt. Any output that would normally be printed to the console will be written to the file instead.

    This can be useful for saving the output of a curl command for later analysis or processing. For example, you could use the output file as input for a script or program that parses the HTTP response and extracts specific data.

    You can also append the output to an existing file by using >> instead of >. For example:

    curl https://example.com >> output.txt
    This command will append the output of curl to the end of the file output.txt. If the file doesn't exist, it will be created.

  5. Use the --trace flag: The --trace flag allows you to see the entire HTTP request and response, including headers, in a file. This can be especially useful for debugging complex issues. Example:

    curl --trace trace.txt https://example.com

    In this example, the --trace flag is used to generate a trace file called trace.txt. The trace file will contain detailed information about the request and response, including the headers and any data that is sent or received.

    This can be useful for debugging complex requests or troubleshooting issues with a web application. You can use the trace file to analyze the HTTP traffic and identify any errors or issues with the request.

    By default, the --trace flag will overwrite the contents of the trace file if it already exists. If you want to append to an existing trace file, you can use the --trace-ascii flag instead:

    curl --trace-ascii trace.txt https://example.com
    This command will append the trace information to the end of the file trace.txt, rather than overwriting it.

Examples

  • Sending a GET request:

curl -v https://example.com

This command sends a GET request to https://example.com and displays the response headers and body.

  • Sending a POST request with data payload:

curl -X POST -d "name=John&age=30" https://example.com/user

This command sends a POST request to https://example.com/user with a data payload of name=John and age=30.

  • Sending a request with headers:

curl -H "Authorization: Bearer <access_token>" https://example.com/api/user

This command sends a request to https://example.com/api/user with an Authorization header containing an access token.

  • Saving the response to a file:

curl -o response.txt https://example.com

This command sends a GET request to https://example.com and saves the response to response.txt.

  • Using the --trace flag:

curl --trace trace.log https://example.com

This command sends a GET request to https://example.com and saves the entire HTTP request and response, including headers, to trace.log.

  • Using the --resolve flag:

The --resolve flag in curl allows you to manually specify the IP address or hostname of the server that you want to connect to. This can be useful in scenarios where DNS resolution is not working as expected or you need to test against a specific IP address or hostname.

Here's how to use --resolve:

curl --resolve <hostname>:<port>:<ip_address> https://example.com

In this command, replace <hostname> with the hostname that you want to connect to, <port> with the port number that you want to use (usually 80 or 443 for HTTP and HTTPS respectively), and <ip_address> with the IP address that you want to use to connect to the server.

For example, let's say you have a web application running on a server with an IP address of 10.0.0.10, but the hostname example.com resolves to a different IP address. You can use --resolve to connect to the server directly using its IP address:

curl --resolve example.com:443:10.0.0.10 https://example.com

This command will connect to the server at IP address 10.0.0.10 on port 443, but the HTTP request will contain the hostname example.com.

Note that the --resolve flag is only effective for the hostname and port specified in the URL. If the HTTP request contains any other hostnames, they will still be resolved using DNS. Also, keep in mind that using --resolve can be useful in testing and debugging scenarios, but it should not be used as a permanent solution as it can cause issues with load balancing and other network configurations.

In conclusion, curl is a powerful tool that can be extremely helpful in debugging web applications. By following best practices, using tips and tricks, and utilizing valuable curl scripts, you can save time and make the debugging process much smoother.

Tags

Comments