.png)
On March 13, 2022, OpenAI shocked the world by releasing GPT-4 through a live event streamed on YouTube. The company also released an optimized version of GPT-3.5. Now both large language models (LLMs) and the legacy (previous) LLM are available on ChatGPT. In this post, I introduce the three models currently available on ChatGPT Plus, five prompts that I used to play around with the new models, and the strides made by OpenAI in teaching ChatGPT Mule 4, since my observations on Feb 21, 2023. Check out my previous post on my comments regarding MuleSoft development on GPT 3.5 (now Legacy) model.
Quick Navigation
Legal Disclaimer: This post consists of experiments, conclusions, and opinions that originate with me and are not affiliated with my current employer at the time of this post, Salesforce and MuleSoft.
Differences between OpenAI GPT models
Before I dive deep, these three available models are available in ChatGPT.
Model | Description |
Legacy 3.5 |  |
Default (GPT-3.5) |  |
GPT-4 |  |
First Impressions Video
Check out this embedded first impressions video to view my reaction as I played with the latest ChatGPT release.
Experiment
Prompts
While experimenting with the latest release, I used the following prompts and compared the outputs between GPT 3.5 Optimized and GPT 4. These prompts represent typical tasks done by MuleSoft developers.
Prompt No. | Prompt |
1 | Build a Mule 4 flow that prints “Hello World” |
2 | Generate an MUnit test case for this code |
3* | What Mule 4 scope will allow me to process flows asynchronously? |
4* | Build a Mule 4 flow that listens to an HTTP GET request on port 8082 with a query parameter that takes a name and returns a JSON string of “Hello” concatenated with the value of the “name” query parameter. |
5* | Write the Mule 4 code to enable CSV streaming on an SFTP connector. |
* Prompts with * will not be discussed in this post.
Prompt 1: Build a Mule 4 flow that prints Hello World
Default (GPT 3.5) -Optimized
Code
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
<http:listener-config name="HTTP_Listener_config" host="localhost" port="8081" doc:name="HTTP Listener config"/>
<flow name="hello-world">
<http:listener path="/" config-ref="HTTP_Listener_config" doc:name="HTTP"/>
<set-payload value="Hello World" doc:name="Set Payload"/>
<logger level="INFO" doc:name="Logger" message="#[payload]"/>
</flow>
</mule>Observations
Notice that the default version can give you steps to accomplish your task. The output is different from the legacy version. However, it’s interesting that the code uses a “Set-Payload” component over a “Transform Message” component. What could this mean? Could it be that the model still learning DataWeave?
Also, this code provides an entire XML file, instead of a code snippet, unlike the legacy model. And lastly, the code snippet no longer says php, unlike the legacy model that labeled the code as PHP during my February 21st experiment.
GPT-4
Now this gets interesting. GPT 4 provides steps like the optimized version of GPT 3.5. But wait? Where is the code?
Code
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd">
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config">
<http:listener-connection host="0.0.0.0" port="8081" />
</http:listener-config>
<flow name="HelloWorldFlow" >
<http:listener doc:name="Listener" config-ref="HTTP_Listener_config" path="/hello"/>
<set-payload value="Hello World" doc:name="Set Payload" />
<logger level="INFO" doc:name="Logger" message="#[payload]" />
</flow>
</mule>Observations
The first prompt did not provide any code, so I had to give a second prompt. I told ChatGPT to provide me with the code. This took two steps. To prevent having to ask for a code snippet in a subsequent step explicitly, I could have combined the two prompts by explicitly asking ChatGPT to include the code in its response.
Prompt 2: Generate an MUnit test case for this code
Default (GPT 3.5) - Optimized
Code
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:mock="http://www.mulesoft.org/schema/mule/mock"
xmlns:munit="http://www.mulesoft.org/schema/mule/munit"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/mock http://www.mulesoft.org/schema/mule/mock/current/mule-mock.xsd
http://www.mulesoft.org/schema/mule/munit http://www.mulesoft.org/schema/mule/munit/current/mule-munit.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
<http:listener-config name="HTTP_Listener_config" host="localhost" port="8081" doc:name="HTTP Listener config"/>
<flow name="hello-world">
<http:listener path="/" config-ref="HTTP_Listener_config" doc:name="HTTP"/>
<set-payload value="Hello World" doc:name="Set Payload"/>
<logger level="INFO" doc:name="Logger" message="#[payload]"/>
</flow>
<munit:test name="hello-world-test" description="Test the hello-world flow">
<mock:when messageProcessor="http:listener" doc:name="Mock HTTP Listener">
<mock:then-return payload="Hello World"/>
</mock:when>
<set-payload value="Hello World" doc:name="Set Payload"/>
<logger level="INFO" doc:name="Logger" message="#[payload]"/>
<munit:assert-true message="The payload should be 'Hello World'">
<munit:expression>#['Hello World' as String = payload as String]</munit:expression>
</munit:assert-true>
</munit:test>
</mule>Observations
At first glance, GPT 3.5 (Default) - optimized seems to understand how to create a general MUnit test case for the first prompt. However, this expression is incorrect.
<munit:expression>#['Hello World' as String = payload as String]</munit:expression>Can you spot the bug? If so, comment below.
GPT-4
Code
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:munit="http://www.mulesoft.org/schema/mule/munit"
xmlns:munit-tools="http://www.mulesoft.org/schema/mule/munit-tools"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/munit http://www.mulesoft.org/schema/mule/munit/current/mule-munit.xsd
http://www.mulesoft.org/schema/mule/munit-tools http://www.mulesoft.org/schema/mule/munit-tools/current/mule-munit-tools.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd">
<munit:config name="hello-world-flow-test" />
<http:request-config name="HTTP_Request_Configuration" >
<http:request-connection host="localhost" port="8081" />
</http:request-config>
<munit:test name="helloWorldFlowTest" description="Test Hello World flow">
<munit:execution>
<http:request method="GET" path="/hello" config-ref="HTTP_Request_Configuration" />
</munit:execution>
<munit:validation>
<munit-tools:assert-that expression="#[payload]" is='#[MunitTools::equalTo("Hello World")]' />
<munit-tools:assert-that expression="#[attributes.statusCode]" is='#[MunitTools::equalTo(200)]' />
</munit:validation>
</munit:test>
</mule>Observations
GPT 4 provided a slightly different code. For instance, GPT 4 chose not to use a mock when event processor, unlike Default (GPT3.5) - optimized. Instead, it used assert-that event processors.
Another observation was GPT-4 was able to separate the MUnit code by execution and validation. Default (GPT3.5) - optimized did not.
Prompt Engineering
After watching loads of YouTube videos and playing with different generative AI applications, I have found that there is a science to getting the best response. Users of these models must make good prompts. This is so important that a new engineering discipline is forming. It's called prompt engineering. :) Once I get my hands on the next release, I plan to use prompt engineering techniques to improve the responses further.
Additional Comments
Overall, I see a significant improvement from Legacy GPT-3.5 to the Default (GPT-3.5) - optimized and GPT- 4 LLMs. Here are a few.
As promised by OpenAI, both GPT versions received a significant boost to speed.
Both models now produce detailed steps before providing code. This will help you to understand the model's problem-solving and reasoning process.
The detailed steps also show the two model's reasoning has improved. I want to ask the models in my next experiment why they chose specific responses.
Should MuleSoft developers fear ChatGPT? No. However, with Einstein GPT, it would not surprise me if a GPT version is released in an Anypoint Platform product. If that is the case, you should continue learning Mule 4 and DataWeave so you can provide good prompts to a co-pilot tool if such a tool should be released.
Did you like this post? If so, please consider joining my site by filling out the subscription form in the footer and leaving a like or comment below. You will be the first to know when I release part 2 of this post next week.
留言