pulumi/sdk/python/lib/test/automation/test_stack.py

65 lines
2.2 KiB
Python
Raw Normal View History

automation: only read complete lines before trying to deserialize (#15778) When tailing the event log in automation API we currently have nothing that makes sure we read only complete lines. This means if the OS happens to flush an incomplete line for whatever reason (or the Go JSON encoder does, which we're using to write these lines), we might read a line that is incompletely written, and thus will fail to JSON decode it. Since the JSON encoder always writes a newline at the end of each string, we can also make sure that the line we read ends with a newline and otherwise wait for the rest of the line to be written. The library we use in Go provides a convenient setting for this, while in python and nodejs we need to add some code to do this ourselves. Fixes https://github.com/pulumi/pulumi/issues/15235 Fixes https://github.com/pulumi/pulumi/issues/15652 Fixes https://github.com/pulumi/pulumi/issues/9269 (This is closed already, but never had a proper resolution afaics) Fixes https://github.com/pulumi/pulumi/issues/6768 It would be nice to add a typescript test here as well, but I'm not sure how to do that without marking the readLines function non-private. But I don't know typescript well, so any hints of how to do that would be appreciated! ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-26 14:32:56 +00:00
# Copyright 2024-2024, Pulumi Corporation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import asyncio
import os
import tempfile
import time
import unittest
from pulumi.automation._stack import _watch_logs
from pulumi.automation import EngineEvent, StdoutEngineEvent
automation: only read complete lines before trying to deserialize (#15778) When tailing the event log in automation API we currently have nothing that makes sure we read only complete lines. This means if the OS happens to flush an incomplete line for whatever reason (or the Go JSON encoder does, which we're using to write these lines), we might read a line that is incompletely written, and thus will fail to JSON decode it. Since the JSON encoder always writes a newline at the end of each string, we can also make sure that the line we read ends with a newline and otherwise wait for the rest of the line to be written. The library we use in Go provides a convenient setting for this, while in python and nodejs we need to add some code to do this ourselves. Fixes https://github.com/pulumi/pulumi/issues/15235 Fixes https://github.com/pulumi/pulumi/issues/15652 Fixes https://github.com/pulumi/pulumi/issues/9269 (This is closed already, but never had a proper resolution afaics) Fixes https://github.com/pulumi/pulumi/issues/6768 It would be nice to add a typescript test here as well, but I'm not sure how to do that without marking the readLines function non-private. But I don't know typescript well, so any hints of how to do that would be appreciated! ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-26 14:32:56 +00:00
class TestStack(unittest.IsolatedAsyncioTestCase):
async def test_always_read_complete_lines(self):
tmp = tempfile.NamedTemporaryFile(delete=False)
async def write_lines(tmp):
with open(tmp, "w") as f:
parts = [
'{"stdoutEvent": ',
'{"message": "hello", "color": "blue"}',
"}\n",
'{"stdoutEvent": ',
'{"message": "world"',
', "color": "red"}}\n',
'{"cancelEvent": {}}\n',
]
automation: only read complete lines before trying to deserialize (#15778) When tailing the event log in automation API we currently have nothing that makes sure we read only complete lines. This means if the OS happens to flush an incomplete line for whatever reason (or the Go JSON encoder does, which we're using to write these lines), we might read a line that is incompletely written, and thus will fail to JSON decode it. Since the JSON encoder always writes a newline at the end of each string, we can also make sure that the line we read ends with a newline and otherwise wait for the rest of the line to be written. The library we use in Go provides a convenient setting for this, while in python and nodejs we need to add some code to do this ourselves. Fixes https://github.com/pulumi/pulumi/issues/15235 Fixes https://github.com/pulumi/pulumi/issues/15652 Fixes https://github.com/pulumi/pulumi/issues/9269 (This is closed already, but never had a proper resolution afaics) Fixes https://github.com/pulumi/pulumi/issues/6768 It would be nice to add a typescript test here as well, but I'm not sure how to do that without marking the readLines function non-private. But I don't know typescript well, so any hints of how to do that would be appreciated! ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-26 14:32:56 +00:00
for part in parts:
f.write(part)
f.flush()
time.sleep(0.2)
write_task = asyncio.create_task(write_lines(tmp.name))
event_num = 0
def callback(event):
nonlocal event_num
if event_num == 0:
self.assertTrue(event.stdout_event)
self.assertEqual(event.stdout_event.message, "hello")
self.assertEqual(event.stdout_event.color, "blue")
elif event_num == 1:
self.assertTrue(event.stdout_event)
self.assertEqual(event.stdout_event.message, "world")
self.assertEqual(event.stdout_event.color, "red")
event_num += 1
async def watch_async():
_watch_logs(tmp.name, callback)
automation: only read complete lines before trying to deserialize (#15778) When tailing the event log in automation API we currently have nothing that makes sure we read only complete lines. This means if the OS happens to flush an incomplete line for whatever reason (or the Go JSON encoder does, which we're using to write these lines), we might read a line that is incompletely written, and thus will fail to JSON decode it. Since the JSON encoder always writes a newline at the end of each string, we can also make sure that the line we read ends with a newline and otherwise wait for the rest of the line to be written. The library we use in Go provides a convenient setting for this, while in python and nodejs we need to add some code to do this ourselves. Fixes https://github.com/pulumi/pulumi/issues/15235 Fixes https://github.com/pulumi/pulumi/issues/15652 Fixes https://github.com/pulumi/pulumi/issues/9269 (This is closed already, but never had a proper resolution afaics) Fixes https://github.com/pulumi/pulumi/issues/6768 It would be nice to add a typescript test here as well, but I'm not sure how to do that without marking the readLines function non-private. But I don't know typescript well, so any hints of how to do that would be appreciated! ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-26 14:32:56 +00:00
watch_task = asyncio.create_task(watch_async())
await asyncio.gather(write_task, watch_task)