Blog

I wanted to compare the two tools to see how they differ in their responses. I gave it a prompt to get more information about AI developer tools. I was using ChatGPT version 3.5 and whatever Bing Chat is running today. Keep in mind that ChatGPT only has data up to September 2021, while Bing Chat can query the internet in real-time. The prompt I used for each of them is:

compare and contrast chatgpt vs github copilot for aiding in software development. focus on ease of use, accuracy, and ability to handle complex coding requirements

The result I got from ChatGPT:

Both ChatGPT and GitHub Copilot aim to aid in software development, but they differ in their approach, underlying technologies, and capabilities. Let’s compare and contrast them in terms of ease of use, accuracy, and ability to handle complex coding requirements:

Ease of Use:

• ChatGPT: ChatGPT provides a conversational interface where you can interactively ask questions, seek guidance, or request code snippets. It leverages natural language understanding and generation to provide responses. It offers a familiar chat-based experience.
• GitHub Copilot: GitHub Copilot is integrated directly into code editors, such as Visual Studio Code. It suggests code completions, whole lines, or even entire functions based on the context. It seamlessly integrates into the development workflow.

Accuracy:

• ChatGPT: ChatGPT’s responses are generated based on pre-trained language models. While it can offer helpful insights, it might not always provide code that’s syntactically correct or optimized. It heavily relies on the quality of the training data and may occasionally produce inaccurate or irrelevant responses.
• GitHub Copilot: GitHub Copilot utilizes OpenAI’s Codex, which is trained on a large corpus of publicly available code. It can often provide accurate and syntactically correct suggestions due to its exposure to vast amounts of real-world code. However, it can still generate imperfect code that may require manual review.

Ability to Handle Complex Coding Requirements:

• ChatGPT: ChatGPT can assist with a wide range of coding requirements, from general programming concepts to specific language queries. However, it may struggle with highly specialized or domain-specific topics and might not be able to handle complex coding requirements as effectively.
• GitHub Copilot: GitHub Copilot has an advantage when it comes to handling complex coding requirements. It excels at generating code for various programming languages, frameworks, and libraries. With access to extensive code repositories, it can handle a wide array of coding tasks and provide relevant suggestions.

In summary, both ChatGPT and GitHub Copilot have their strengths and limitations. ChatGPT offers a conversational interface and can provide general guidance, while GitHub Copilot is tightly integrated with code editors and can suggest accurate code snippets. GitHub Copilot is particularly well-suited for handling complex coding requirements due to its exposure to vast code repositories. However, both tools may occasionally generate imperfect or incomplete code that requires manual review.

And this is what I got from Bing Chat:

ChatGPT and GitHub Copilot are both generative AI tools that can assist coders in application development, but they have different strengths and weaknesses. Here are some points of comparison based on the search results²³⁴:

• Purpose: GitHub Copilot was designed with code generation and completion in mind, while ChatGPT acts as a general-purpose language model that can be utilized for many language-related tasks¹.
• Output: GitHub Copilot generates code snippets as output, while ChatGPT produces natural language responses¹. This means that Copilot can help you write code faster and more accurately, while ChatGPT can help you understand code better and provide explanations or summaries.
• Integration: GitHub Copilot is tightly integrated with popular IDEs such as Visual Studio Code, Visual Studio, Neovim, and JetBrains IDEs², while ChatGPT is accessed as a separate tool. This means that Copilot can analyze a large context of code without having to cut and paste short snippets into ChatGPT.
• Learning: GitHub Copilot uses machine learning to constantly learn from code and behavior, improving suggestions over time³, while ChatGPT can help generate code and follow up on it in a particular conversation, but once the conversation is lost, it cannot continue without a special prompt³.
• Languages: GitHub Copilot supports a wide range of programming languages, such as Python, JavaScript, TypeScript, Ruby, Go, PHP, Swift, Kotlin, Rust, C#, C++, Java, HTML/CSS, SQL³, while ChatGPT supports fewer languages, such as Python, Java, JavaScript, C++, Ruby, PHP, Swift, Kotlin, Rust, TypeScript³.

In summary, GitHub Copilot is more suitable for professional developers who need a fast and accurate code completion tool that can work with various languages and IDEs. ChatGPT is more suitable for beginners or learners who need a general language model that can help generate code with explanations or summaries. Both tools have their own advantages and limitations depending on the use case and preference of the user.

Source: Conversation with Bing, 6/2/2023
(1) GitHub Copilot vs. ChatGPT: How do they compare? – TechTarget. https://www.techtarget.com/searchenterpriseai/tip/GitHub-Copilot-vs-ChatGPT-How-do-they-compare.
(2) Github Copilot vs ChatGPT: Basic Differences to Know – Nerds Chalk. https://nerdschalk.com/github-copilot-vs-chatgpt/.
(3) ChatGPT vs GitHub Copilot [Real Code Comparison] – EML. https://enjoymachinelearning.com/blog/chatgpt-vs-github-copilot/.
(4) GitHub Copilot vs ChatGPT: What is the Main Difference. https://www.weetechsolution.com/blog/github-copilot-vs-chatgpt.

My thoughts…

While the results are similar and both are useful, I think it’s interesting to note that Bing included links. In the actual Edge browser, the text it provided linked directly to a source website, so visiting those is extremely easy. When I copy/pasted the content, the links showed up as footnotes, which is also kind of neat.

Notice that ChatGPT was more verbose, and Bing Chat was more specific, such as listing specific languages that each supports. Also, ChatGPT answered the prompt in the exact format asked, while Bing Chat expanded the items it listed in the response.