별집사의 IT세상

[Kafka] kafka connect transform issue 정리

프로젝트 시 issue를 계속 추가할 예정. 특정 필드에 타임스탬프 시간 insert "transforms": "InsertField", "transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value", "transforms.InsertField.timestamp.field": "create_dtm", debizium connector timestamp issue debizium source connector에서 timestamp를 연결할 시 ZonedTimedStamp 형식으로 값이 전환되어 오류가 발생한다. "transforms": "TimestampConverter", "transforms.TimestampC..
IT/Kafka 2022. 9. 7. 18:07
[Kafka] Kafka Config Ackmode

BATCH Commit the offsets of all records returned by the previous poll after they all have been processed by the listener. COUNT Commit pending offsets after ackCount has been exceeded. COUNT_TIME Commit pending offsets after ackCount has been exceeded or after ackTime has elapsed. MANUAL Listener is responsible for acking - use a AcknowledgingMessageListener; acks will be queued and offsets will..
IT/Kafka 2022. 9. 6. 11:12
[JAVA] kafka sub 리스너 자동 실행 방법

@KafkaListener(id = "아이디", topics = "토픽명", groupId = "sampleGroup", containerFactory = "kafkaListenerContainerFactoryForList", autoStartup = "false") 서비스에서 autoStartup을 false로 지정하면된다. 리스너 실행 수동조작방법은 컨트롤러를 따로 생성해서 endpointRegistry 이용! @RestController public class HandleController { @Autowired private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry; @GetMapping(path = "/start") public ..
IT/Kafka 2022. 9. 5. 10:27
Prev 1 2 3 4 5 6 7 8 9 Next